- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Variable Codebook Size Quantization (VCQ) for autoregressive visual generation. Motivated by a rapid decline in empirical conditional entropy that the authors call the “Entropy Cliff,” VCQ uses a shared codebook and progressively increases the number of entries accessible at each sequence position. Experiments on ImageNet show improved generation quality at a reconstruction cost.
Critical strengths
-
Improved generation quality. With the same baseline training setup, VCQ reduces unguided gFID from 27.98 to 14.80. VCQ-Linear also improves generation over a constant 8192-entry codebook while achieving similar reconstruction quality.
-
Simple implementation. Each position selects from the first K_t entries of a shared codebook. The method adds no parameters and retains the standard tokenizer and AR training objectives, making it easy to incorporate into existing pipelines.
Critical weaknesses
-
The entropy cliff does not establish a failure of AR modeling. Proposition 1 concerns a finite empirical dataset. Unique prefixes have fixed observed continuations, but this says nothing about population-level conditional entropy. To take a concrete case: independent uniform tokens over 16384 codes can produce mostly unique two-token prefixes on ImageNet-sized data while retaining 14 bits of population conditional entropy at every position. Figure 2 and Table 11 document prefix sparsity, not a failure of generalization.
-
The memorization claim lacks model-side evidence. The paper reports predictive entropy averaging 13.5 bits over positions 3-16, never below 9.1 bits. This is hard to reconcile with Figure 1’s Generalize/Memorize framing, and the paper does not really address the tension. What would be needed is a per-position train/val NLL comparison showing that later positions actually overfit.
-
The generation gains are not tied to the proposed mechanism. Changing the codebook schedule jointly changes capacity, reconstruction distortion, and encoder representations, and the ablations do not separate these effects. It is also worth noting that Figure 2 uses a Power schedule rather than the default Cosine, so whether the measured cliff location actually explains the generation quality differences across schedules is never tested.
-
The semantic probing results do not show what information reaches the AR model. Table 5 measures pre-quantization features, so it is unclear whether the semantic information survives quantization. And if probing accuracy were the right proxy, Power should win: it has the highest early-probe accuracy among all schedules but worse unguided gFID than both Linear and Cosine.
Questions for authors
Figure 2(a) is central to the entropy-cliff argument, so I have several questions about how the curves were computed and what they actually show.
-
Were the image curves computed entirely from empirical prefix frequencies using Eq. (4)? It would help to see the plotting code and the numerical per-position conditional entropies and remaining budgets.
-
Could the authors plot the capacity floors alongside the empirical curves with separate labels? The floors depend only on sample count and codebook size, while the measured remaining budgets depend on the observed token distribution. How closely do they agree in practice?
-
Singleton prefixes necessarily have zero empirical continuation entropy, even when population conditional entropy remains high. What evidence shows that the decline in Figure 2(a) reflects impaired AR generalization rather than just finite-sample prefix sparsity?
3 comments
Class information before quantization
reviewer (w4) : The semantic probing results do not show what information reaches the AR model. Table 5 measures pre-quantization features, so it is unclear whether the semantic information survives quantization. And if probing accuracy were the right proxy, Power should win: it has the highest early-probe accuracy among all schedules but worse unguided gFID than both Linear and Cosine.
Using pre-quantization features is standard practice. We do not consider an additional post-quantization linear probe essential evidence for our claim. We pool the early token positions to see how much global information they encode together. Power leads at k=10, Cosine at k=256. Cosine’s early gain over Constant suggests a possible coarse-to-fine structure. Power’s lead on the early probe may be intuitive. Another possibility is that the larger differences in codebook size across positions cause more information to be lost when the early positions are pooled. Power’s result does not undermine the rationale for our probe, the claim it tests, or the effectiveness of VCQ.
Empirical budgets and capacity floors
reviewer (q1) : Were the image curves computed entirely from empirical prefix frequencies using Eq. (4)? It would help to see the plotting code and the numerical per-position conditional entropies and remaining budgets.
reviewer (q2) : Could the authors plot the capacity floors alongside the empirical curves with separate labels? The floors depend only on sample count and codebook size, while the measured remaining budgets depend on the observed token distribution. How closely do they agree in practice?
Figure 2(a) uses standard prefix-frequency counts on the ImageNet validation set (§2.2, Appendix E). Each tokenizer produces one sequence per image. In a prefix tree, a node counts its prefix and its children count the continuations. Eq. (4) averages next-token entropy at each depth, weighted by prefix frequency. Subtracting these entropies from log₂ N gives the remaining budget. The floor uses only N and the codebook sizes.
At the first few positions in Figure 2(a), the measured remaining entropy budget for Constant-16K image sequences closely follows its theoretical lower bound, much more closely than the LAMBADA language curve follows its own.
Follow-up on the author response
Thanks for the detailed rebuttal. The Constant-8K vs. VCQ-Linear comparison is helpful, but most of my concerns are still not resolved. There is still no model-side evidence (e.g., per-position train/val NLL) showing that the entropy cliff actually leads to memorization. Right now the results can only be explained by finite-sample prefix sparsity. The link between cliff location and generation quality is also not tested. And I didn’t see the Figure 2(a) plotting code or the comparison with the capacity floors that I asked for. So I’ll keep my original score.
Thank you for reviewing our work and recognizing VCQ’s simplicity and generation gains. We address your questions below.
High predictive entropy after the empirical cliff
Your example assumes away the very dependence at issue. With independent positions, shuffling prefixes leaves the optimal next-token prediction unchanged. Image tokens depend on one another, so the model must use the prefix. Yet with Constant-16K, two tokens already identify about 97% of training images (Table 11). A model can then recognize the prefix, memorize the suffix, and predict every remaining training token correctly. An example that makes the prefix irrelevant cannot rule out memorization through that very prefix.
Consider a simple AR model with unlimited capacity, little weight decay and little parameter sharing. Once a training prefix is unique, the empirical optimum at that prefix amounts to memorizing its observed continuation. Finite capacity, weight decay and shared parameters help a practical AR model avoid memorization to some extent. The data-side problem persists. We asked whether improving the data representation under those same conditions would still improve generation. Our results support that hypothesis.
Section 2.2 explains this tension in the very passage you cite. The paragraph reporting 13.5 bits of mean validation predictive entropy at positions 3–16 explains regularization and cross-prefix sharing. The next gives the controlled result. With Base AR training, weight decay (0.03) and unguided sampling temperature (0.85) fixed, switching from Constant-16K to VCQ-Cosine improves unguided gFID from 27.98 to 14.80 (Table 3).
Generation at nearly equal capacity
Constant-8K and VCQ-Linear have nearly identical nominal capacity and similar rFID (2.36 vs. 2.42). Under the same Base training and sampling, Linear improves unguided gFID from 18.16 to 14.70. Figure 2(a) shows Power delaying the cliff (16 initial codes). In Table 3 (2 initial codes), Cosine leads with CFG. Unguided gFID: Linear 14.70, Cosine 14.80, Power 16.19.