acceptodds
Withdrawn from ICLR 2027

How Much of Your Codebook Survives Denoising?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Two-stage image generation compresses an image into discrete codewords—a tokenizer with codebook size K—then models the code sequences. How large should K be? Reported optima span two orders of magnitude. We treat the trained second-stage generator as a measurable noisy channel: renoising and re-denoising real latents yields its per-token confusion kernel, and a surrogate computed from the codebook alone reproduces the measurement within its validity domain. Three rates must be distinguished—nominal (log₂K), used (H(q)), and transferred through the probe—and three findings follow. Snapping: diffusion outputs land exactly on the codeword grid, so generation error means committing to the wrong codeword, invisible to dispersion-based diagnostics. Saturation: across CIFAR and ImageNet (64–256px) the nominal rate keeps growing while probe transfer stalls; in the converged CIFAR-10 sweep, about 8 of 15 nominal bits survive our stress probe. The saturation point, computed from the codebook alone, bounds the useful range of K before any stage-2 is trained: the best gFID inside this range matches the global best on both CIFAR variants and trails it by 0.1 on ImageNet. Allocation: in our matched-budget experiments, the catastrophic large-K collapse of autoregressive models is driven by vocabulary crowding out core parameters rather than by K itself, while diffusion failures trace to stage-2 commitment capacity or codebook geometry—interventions on each remove them. All numerical predictions were archived before measurement; five falsified predictions are reported.

Primary area:
generative models
Keywords:
image tokenization, vector quantization, codebook size, two-stage image generation, latent diffusion, autoregressive image generation, noisy channel, information-theoretic analysis, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2eZhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how VQ codebook size affects two-stage image generation. It tests how well the model recovers tokens after adding noise, and proposes a cheap codebook-only version of this test. It then uses this heuristic to choose an upper limit on the codebook sizes worth trying.

Critical strengths

S1. The paper contains many experiments and controls across datasets, codebook sizes, and model sizes.

Critical weaknesses

W1. I do not see a convincing connection between the synthetic denoising test and generation. The paper measures recovery after artificially corrupting real latents, but this is not the process used during generation. The paper directly tests whether its synthetic renoise–denoise probe matches what happens during actual sampling, and finds that it does not. The codebook-size results provide some empirical support, but I do not see a clear mechanistic explanation of why token recovery in this test should predict unconditional generation quality.

W2. The practical payoff of the codebook-size heuristic appears limited and dependent on generator capacity. On ImageNet-64, it reduces the search from eight codebook sizes to seven. On ImageNet-256, increasing the diffusion model from 21M to 357M parameters reduces the large-codebook gFID penalty from 11.6 to 2.3. The paper acknowledges this capacity effect, but does not show whether its codebook-only upper bound remains useful when generator capacity is varied.

Questions for authors

Q1. What is the intuitive or formal link between recovery in this synthetic denoising test and unconditional generation quality?

Q2. What benefit does VQ provide over continuous latents in this setting, and does that benefit justify studying codebook size?

Reviewer_j8H7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies discrete image generation using a data-conditioned denoising probe. It measures codeword confusion, transferred information, and terminal residuals, comparing them with a codebook-only AWGN nearest-neighbor surrogate. It distinguishes nominal, used, and probe-transferred rates, investigates codebook-size saturation, and examines generator capacity and AR vocabulary/core allocation. Its practical claim is that codebook stress curves can narrow the useful codebook-size search before training stage two.

Critical strengths
  1. A small terminal residual can still accompany recovery of the wrong codeword, which makes the paper's separation of these quantities useful. It also tests how the probe differs from unconditional sampling. The Gaussian-covariance bound is correctly restricted to the memoryless surrogate. These distinctions give the measurement framework a sound basis without overstating what the bound covers.

  2. The repeated-corruption estimates and shuffled-context controls make the observations more convincing by addressing bias and contextual effects. FSQ and the 256px/357M experiments broaden the evidence beyond the main CIFAR sweep, while the fixed-K AR results show that increasing core capacity can substantially repair generation. The reported failures help establish the diagnostics' limits. This empirical work keeps the overall assessment near the borderline despite the concerns below.

Critical weaknesses
  1. An approximately constant error rate PeP_e in Figure 5 does not seem sufficient to exclude wrong commitments as the cause. Figure 2 specifies D(Q(z)): with the decoder and final discrete sequence fixed, within-cell residual changes do not directly change the output. But equal P_e does not fix erroneous codeword identities, displacement, or spatial structure. Dispersion could still affect assignments and remain a useful diagnostic. The paper already analyzes displacement and acknowledges that Pe/AP_e/A alone do not determine damage; its severe-damage example is outside the stated validity domain. The concern is specifically the exclusion claimed for this step-count control.

  2. The proposed rule chooses the first K within 0.25 bits of the maximum surrogate transfer across an available tokenizer sweep, which makes it a retrospective heuristic rather than an established useful-K upper bound. The covariance bound does not bound generation-optimal K, and the paper itself shows that the stress probe differs from the sampler and does not measure perceptual quality. This practical claim needs evidence on configurations not used to develop the rule, including selection regret and uncertainty. Together with the concern about excluding assignment changes as the cause, this leaves the assessment near the borderline, leaning toward rejection.

Questions for authors
  1. How do erroneous codeword identities, displacement, and spatial error structure change across the step-count sweep? Would these measurements support excluding assignment changes, or should the claim be narrowed? The existing displacement and boundary analyses do not establish this particular exclusion.

  2. Please report retained-search regret and compute savings on held-out configurations, with the K-selection rule fixed before evaluating generation quality. A comparison with simpler codebook/reconstruction statistics or the already discussed pFID diagnostic would help assess the pre-training selection claim.

Reviewer_EnHqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem of undetermined codebook size K and the reconstructiongeneration tradeoff in twostage image generation by treating the secondstage generator as a noisy channel. Through roundtrip stress tests, it distinguishes three rates: nominal, used, and transferred. It finds that the transferred rate saturates in advance, and a large number of nominal bits are lost during denoising. The paper reveals the snapping phenomenon of diffusion models and points out that traditional residualbased diagnostics overlook codeword commitment errors. It proposes an AWGN surrogate that relies solely on the codebook, which can determine a reasonable upper bound for K without training any secondstage model. Meanwhile, it analyzes the root causes of performance degradation: degradations in diffusion models stem from stage2 commitment capacity or codebook geometry issues, whereas autoregressive model failures arise from vocabulary crowding out core parameters. This provides diagnostic tools for codebooksize selection.

Critical strengths

This work designs an AWGN surrogate relying only on codebook statistics. Without training the computationally expensive second‑stage generator model, it can estimate effective transferred information and pre‑specify the upper bound for codebook‑size K search, thus cutting hyperparameter‑tuning costs. Furthermore, it clarifies the internal mechanisms behind generation‑performance degradation induced by large K, breaking the conventional perception that blames K per se for such issues. Through controlled comparative experiments, it differentiates the distinct failure origins of diffusion models and autoregressive models and delivers targeted remediation directions. It deepens the mechanistic understanding of the reconstruction‑generation trade‑off and supplies practical information‑theoretic diagnostic insights for discrete‑image tokenizer selection.

Critical weaknesses

The paper has the following weaknesses: 1.The core diagnostic tool of this paper relies on the conditional roundtrip stress probe. This probe channel is not equivalent to the model’s native unconditional sampling channel. Under moderate noise, the probe overestimates the token error rate. However, lacking quantitative correction methods, the paper directly uses the transferred bits obtained from the probe for preselecting codebook size K. This casts doubt on the practical usability of its key contribution, namely determining the upper bound of K without training the secondstage model. 2.This paper generalizes the parameterallocation effect observed under specific experiments to explain largecodebook degradation in VQAR. It fails to address the inconsistency reported in some literature where performance degradation still occurs with large K even with adequate parameter configurations. Meanwhile, complete experiments are not conducted on mainstream masked generative models, yet the analytical conclusions are regarded as universal laws for twostage discrete image generation.

Questions for authors

I have several questions that require clarification from the authors: 1.The contextual gain Δctx introduces prediction bias, yet no practical compensation scheme is provided. As a result, the codebookbased prefiltering tool is only applicable to small 8×8 grids and is hard to generalize to realworld highresolution largegrid production scenarios. 2.The token error rate obtained from the roundtrip stress probe is substantially higher than that from native unconditional generation trajectories. Could you quantify the bias of the probederived metric Reff relative to the true informationtransfer capacity during unconditional generation? What evidence supports that your proposed Kselection rule remains reliable despite the mismatch between the probe channel and the real generation channel? 3.Prediction P2′ is partially falsified under extremely largeK conditions. The two explanations for this failure are both posthoc interpretations formulated after observing experimental results. Are there any preregistered measurable metrics that can predict the failure of P2′ in advance? How much risk does such posthoc reasoning introduce to the codebookonly Kselection pipeline? 4.Several falsified preregistered predictions are only documented in appendices. Posthoc rationalizations are adopted without quantitative evaluation of their negative impacts on the core codebook prefiltering rules, which weakens the persuasiveness of the preregistration validation paradigm. 5.Although the research motivation is clearly stated, the main body of the manuscript suffers from disorganized writing. Theory and experimental sections are interleaved, raising the barrier for readers to understand the core methodology.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.