- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the semantic structure of a visual tokenizer affects downstream diffusion training. The main observation is that predicting semantics directly from a noisy latent is not the same as decoding the posterior-mean clean latent and then extracting its semantics. The authors call the desired agreement between these two quantities Semantic Affine Consistency, or SAC, and give an error decomposition to motivate it.
Based on this observation, the paper introduces MSAC, a tokenizer-side metric based on interpolation between neighboring latents. It also proposes AffineTok, which contains two training-time components. GSCT uses a globally supervised token to make semantic organization more consistent across images, while PMSA supervises the semantics recovered from an estimated posterior-mean latent. On ImageNet 256×256, AffineTok improves over the RecTok baseline and reports 1.21 gFID without guidance and 1.10 with guidance.
Critical strengths
-
The paper points out a meaningful gap in existing semantic supervision. Prior methods mainly ask whether clean-image semantics can be predicted from a noisy latent, while the actual diffusion pipeline predicts a clean latent first and then decodes it. The paper explains this distinction clearly and connects it to a specific term in the semantic recovery error.
-
The method is relatively easy to use in practice. GSCT and PMSA are only needed during tokenizer training, so the final tokenizer interface and the downstream diffusion training procedure remain unchanged. The improvement over RecTok under the same downstream setting is also encouraging.
Critical weaknesses
-
My main concern is the narrow experimental setting. The actual AffineTok training method is only evaluated on top of RecTok, mainly for class-conditional ImageNet generation at 256×256 with one downstream model. The results therefore do not show whether GSCT and PMSA also help other tokenizer families, compression settings, datasets, or diffusion objectives. The cross-tokenizer experiments validate MSAC as a metric, but they do not validate AffineTok as a general training method.
-
The evidence for the proposed mechanism could be stronger. MSAC approximates the diffusion posterior using the midpoint between neighboring clean latents, but it is unclear how closely this matches the posterior means encountered during actual diffusion. The correlation experiment also uses only 13–14 tokenizer checkpoints. Confidence intervals or leave-one-family-out results would help show that the high correlation is not driven by a few tokenizer families or outliers. In addition, the component ablation is only reported at 20 epochs, so it is hard to tell whether GSCT and PMSA improve final performance or mainly speed up early training.
Questions for authors
1.Could the authors report the GSCT-only and PMSA-only results after longer downstream training, rather than only at 20 epochs? This would help determine whether both modules improve the final result or mainly accelerate convergence.
2.Have the authors tested AffineTok on a tokenizer other than RecTok, or with a different downstream diffusion or flow model? Even one additional substantially different setting would make the generality claim more convincing.
3.How stable is the MSAC correlation under bootstrap resampling or leave-one-tokenizer-family-out evaluation? It would also be useful to see how the metric behaves at different interpolation coefficients and noise levels.
4.What is the additional training cost of PMSA, including its parameter count, memory usage, and training time?