acceptodds
Under review as a conference paper at ICLR 2027

AffineTok: Semantic Affine Consistency for Diffusion-Friendly Visual Tokenizer

Abstract

Visual tokenizers increasingly inject semantic supervision into latent spaces to make downstream diffusion easier. Yet how these semantics should be organized to facilitate denoising remains underexplored. In this paper, we define the semantic recovery objective: the denoising process should recover the semantic content of the ground-truth image from noisy latent, and a good tokenizer should make it easier. Existing approaches train a projector to predict the semantics directly from the noisy latent. We argue that this predicts the average of clean-image semantics, whereas what really needs to be aligned is the semantic of averaged clean latents. More importantly, we demonstrate that the semantic recovery error orthogonally decomposes into the error of the optimal semantic prediction directly from the noisy latent and the error between these two semantic predictions. We therefore identify their consistency as the missing requirement and call it Semantic Affine Consistency (SAC). To examine whether this overlooked requirement is closely related to downstream generation, we introduce Msac, a tokenizer-side proxy for SAC. Across the evaluated tokenizers and diffusion model scales, Msac closely tracks generation quality, reaching a Pearson correlation of with SiT-XL gFID, thereby establishing SAC as a practical diagnostic and motivating SAC-guided tokenizer training. We then introduce AffineTok, which promotes SAC through two complementary, training-only components. Global Semantic Coordination Token(GSCT) coordinates the semantic organization of clean latents, keeping semantic averaging meaningful, while Posterior-Mean Semantic Alignmen (PMSA) predicts posterior-mean latents from noisy inputs and supervises their semantics. On ImageNet \(256\), compared with the baseline, AffineTok reduces gFID by \(26%\) at 20 epochs and, with continued training, achieves a new state-of-the-art gFID of \(1.21\) without classifier-free guidance and 1.10 with guidance.

Primary area:
generative models
Keywords:
vae, diffusability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6xSROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the semantic structure of a visual tokenizer affects downstream diffusion training. The main observation is that predicting semantics directly from a noisy latent is not the same as decoding the posterior-mean clean latent and then extracting its semantics. The authors call the desired agreement between these two quantities Semantic Affine Consistency, or SAC, and give an error decomposition to motivate it.

Based on this observation, the paper introduces MSAC, a tokenizer-side metric based on interpolation between neighboring latents. It also proposes AffineTok, which contains two training-time components. GSCT uses a globally supervised token to make semantic organization more consistent across images, while PMSA supervises the semantics recovered from an estimated posterior-mean latent. On ImageNet 256×256, AffineTok improves over the RecTok baseline and reports 1.21 gFID without guidance and 1.10 with guidance.

Critical strengths
  1. The paper points out a meaningful gap in existing semantic supervision. Prior methods mainly ask whether clean-image semantics can be predicted from a noisy latent, while the actual diffusion pipeline predicts a clean latent first and then decodes it. The paper explains this distinction clearly and connects it to a specific term in the semantic recovery error.

  2. The method is relatively easy to use in practice. GSCT and PMSA are only needed during tokenizer training, so the final tokenizer interface and the downstream diffusion training procedure remain unchanged. The improvement over RecTok under the same downstream setting is also encouraging.

Critical weaknesses
  1. My main concern is the narrow experimental setting. The actual AffineTok training method is only evaluated on top of RecTok, mainly for class-conditional ImageNet generation at 256×256 with one downstream model. The results therefore do not show whether GSCT and PMSA also help other tokenizer families, compression settings, datasets, or diffusion objectives. The cross-tokenizer experiments validate MSAC as a metric, but they do not validate AffineTok as a general training method.

  2. The evidence for the proposed mechanism could be stronger. MSAC approximates the diffusion posterior using the midpoint between neighboring clean latents, but it is unclear how closely this matches the posterior means encountered during actual diffusion. The correlation experiment also uses only 13–14 tokenizer checkpoints. Confidence intervals or leave-one-family-out results would help show that the high correlation is not driven by a few tokenizer families or outliers. In addition, the component ablation is only reported at 20 epochs, so it is hard to tell whether GSCT and PMSA improve final performance or mainly speed up early training.

Questions for authors

1.Could the authors report the GSCT-only and PMSA-only results after longer downstream training, rather than only at 20 epochs? This would help determine whether both modules improve the final result or mainly accelerate convergence.

2.Have the authors tested AffineTok on a tokenizer other than RecTok, or with a different downstream diffusion or flow model? Even one additional substantially different setting would make the generality claim more convincing.

3.How stable is the MSAC correlation under bootstrap resampling or leave-one-tokenizer-family-out evaluation? It would also be useful to see how the metric behaves at different interpolation coefficients and noise levels.

4.What is the additional training cost of PMSA, including its parameter count, memory usage, and training time?

Reviewer_KwYHOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AffineTok, a diffusion-friendly visual tokenizer motivated by Semantic Affine Consistency (SAC). The authors introduce a tokenizer-side metric to characterize latent diffusability and develop two complementary components, Global Semantic Coordination Token (GSCT) and Posterior-Mean Semantic Alignment (PMSA), to improve semantic organization and recovery. Extensive experiments on ImageNet-256 demonstrate promising improvements in generation quality.

Critical strengths

1.The paper provides a well-motivated theoretical analysis of semantic consistency in latent diffusion, offering valuable insights into the relationship between visual tokenization and generation quality.

2.The experimental results are promising, achieving competitive generation performance. The ablation studies further support the effectiveness of the proposed components.

Critical weaknesses

1.Limited evaluation scenarios. The experiments are primarily conducted on ImageNet-256. Evaluating the proposed method on ImageNet-512 and text-to-image (T2I) generation would provide stronger evidence of its generalizability. I would be happy to increase my score if the authors could provide these additional results.

2.Scaling of visual foundation models. The proposed method adopts DINOv3-S as the visual foundation model. It would be interesting to investigate whether larger DINOv3 variants could further improve generation quality. I would be willing to raise my score if the authors could provide additional results on this aspect.

3.Incomplete feature-extractor ablation. The feature-extractor ablation (Table B.1) would be more informative if reconstruction FID (rFID) and generation FID (gFID) were reported for different feature extractors. These additional results would help clarify their influence on reconstruction and generation performance. I would be happy to reconsider my score if the authors could provide a more comprehensive comparison.

4.Missing inference efficiency analysis. Although the proposed components are used only during tokenizer training, the paper would benefit from reporting actual end-to-end inference latency and throughput, with a direct comparison against the baseline under identical experimental settings.

Questions for authors

1.Limited evaluation scenarios.

2.Scaling of visual foundation models.

3.Incomplete feature-extractor ablation.

4.Missing inference efficiency analysis.

If the author can provide a more comprehensive analysis, explanation, and answer to the above content, I would be very willing to raise my score.

Reviewer_qRqoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how visual tokenizers should organize latent space semantics to facilitate denoising and generation in downstream diffusion models. The authors identify a mismatch that existing noisy latent semantic supervision does not explicitly address: the difference between the average of clean image semantics and the semantics decoded from the posterior-mean clean latent. Building on the analysis, the paper introduces a tokenizer-side proxy metric and proposes AffineTok, which combines a Global Semantic Coordination Token (GSCT) with Posterior-Mean Semantic Alignment (PMSA). On ImageNet 256×256, using RecTok as the baseline, DINOv3-S for semantic supervision, and DiTDH-XL as the backbone, AffineTok reduces gFID without classifier-free guidance from 4.44 to 3.28 after 20 epochs of downstream training and reaches 1.21 after 600 epochs. Ablation results support the complementary roles of GSCT and PMSA and suggest that GSCT’s gains are associated with its global semantic supervision rather than merely increasing the number of encoder tokens.

Critical strengths

The paper turns the broad goal of “making the latent space more semantic” into a concrete consistency requirement between two semantic prediction paths. Eq. 6 decomposes the semantic recovery error into the conditional semantic uncertainty given the noisy latent and the discrepancy between the optimal direct semantic prediction and the semantics decoded from the posterior-mean clean latent. This reveals a mismatch between existing supervision and the semantic recovery objective of denoising, providing a clear motivation for SAC and the subsequent method design.

The tokenizer-side diagnostic metric, achieves a Pearson correlation of 0.960 with SiT-XL gFID across 14 tokenizers and outperforms iFID in Pearson, Spearman, and Kendall correlations under both SiT-B and SiT-XL settings. It has the potential to reduce reliance on expensive downstream diffusion training when selecting tokenizers.

Critical weaknesses

The connection between the theoretical objective and the practical optimization remains insufficiently established. Posterior Mean Semantic Alignment relies on a semantic projector calibrated on clean latents to approximate the decoded semantics of posterior mean latents, but the paper does not directly validate this approximation on the distribution of posterior-mean estimates. Moreover, how the theoretical analysis under squared loss relates to the cosine-based semantic loss used in practice requires further discussion. The current results therefore do not clearly establish the extent to which PMSA optimizes the theoretically defined Semantic Affine Consistency .

Although the tokenizer-side SAC metric exhibits strong empirical correlations, its connection to theoretical SAC still relies primarily on a heuristic approximation. Fixed midpoint spherical interpolation between nearest-neighbor latents may not adequately represent the true posterior means at different noise levels. The paper lacks sensitivity analyses for the interpolation method, interpolation coefficient, and neighbor selection, as well as a direct comparison between the proposed metric and SAC errors estimated from diffusion posteriors.

Questions for authors

How accurate is the semantic projector on posterior-mean latent estimates? Since it is calibrated only on clean latents, could you directly compare its predictions with the VFM features extracted from images decoded from posterior-mean estimates, particularly at high noise levels?

How sensitive is the tokenizer-side SAC metric to its interpolation design? Could you evaluate alternative interpolation coefficients, linear versus spherical interpolation, and different neighbor-selection strategies to determine whether its correlation with downstream gFID remains strong?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.