- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how VQ codebook size affects two-stage image generation. It tests how well the model recovers tokens after adding noise, and proposes a cheap codebook-only version of this test. It then uses this heuristic to choose an upper limit on the codebook sizes worth trying.
Critical strengths
S1. The paper contains many experiments and controls across datasets, codebook sizes, and model sizes.
Critical weaknesses
W1. I do not see a convincing connection between the synthetic denoising test and generation. The paper measures recovery after artificially corrupting real latents, but this is not the process used during generation. The paper directly tests whether its synthetic renoise–denoise probe matches what happens during actual sampling, and finds that it does not. The codebook-size results provide some empirical support, but I do not see a clear mechanistic explanation of why token recovery in this test should predict unconditional generation quality.
W2. The practical payoff of the codebook-size heuristic appears limited and dependent on generator capacity. On ImageNet-64, it reduces the search from eight codebook sizes to seven. On ImageNet-256, increasing the diffusion model from 21M to 357M parameters reduces the large-codebook gFID penalty from 11.6 to 2.3. The paper acknowledges this capacity effect, but does not show whether its codebook-only upper bound remains useful when generator capacity is varied.
Questions for authors
Q1. What is the intuitive or formal link between recovery in this synthetic denoising test and unconditional generation quality?
Q2. What benefit does VQ provide over continuous latents in this setting, and does that benefit justify studying codebook size?