- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the mismatch between discrete image tokenizers trained only for reconstruction and the autoregressive (AR) priors later fitted to their tokens. It introduces Tripartite Variational Consistency (TVC), a view of latent-variable learning that separates likelihood, prior and posterior consistency, and uses it to argue that standard two-stage training leaves prior consistency outside the tokenizer objective. To address this, the authors add a distribution-level prior-matching term to tokenizer training and optimise it with a Wasserstein-gradient-flow update, which for discrete tokens is approximated by a per-token contrast between an online AR proxy and a frozen AR reference. The resulting tokenizer, wAR-Tok, is evaluated on CIFAR-10 and ImageNet 256×256, where it lowers AR loss and generation FID at comparable reconstruction quality.
Critical strengths
- The paper works on a relevant problem. Making the tokenizer aware of the prior that will later model its tokens is an open question for AR image generation, and the TVC view states clearly what two-stage training leaves out.
- The final method is simple and cheap. It only needs forward passes through two small AR models and adds about 10% to tokenizer training cost (App. F.3), so it would be easy to adopt if the gains hold up.
- Within the authors' controlled setup, the improvements are consistent. AR loss and gFID go down under both the matched-rFID (Table 1) and matched-iteration (Table 3) protocols, and the ImageNet comparison points in the same direction.
Critical weaknesses
-
My main concern is that the experimental evaluation is far too limited for a tokenizer paper. Sections 2–4, about five pages, are spent on theory and derivation, while the experiments take up roughly half a page. Most results are on CIFAR-10 at 32×32 with 62 tokens per image, where the best gFID is still 11.3–14.5. The ImageNet 256×256 evidence (Table 2) is a single pair of rows, the authors' own 1D-tokenizer baseline and wAR-Tok: gFID improves from 6.32 to 5.42, but rFID gets worse, from 2.05 to 2.35. There is no system-level comparison with published AR tokenizers on ImageNet. This is a notable gap, because the works this paper builds on and compares against (e.g. TiTok, FlexTok, LARP) were themselves evaluated at ImageNet scale, or on standard video benchmarks in the case of LARP, with system-level comparisons; TiTok, for instance, reports a gFID of 1.97 on ImageNet 256×256, far below the 5.42 reported here. Only one tokenizer size and one prior size are used per dataset, so we do not know whether the gain survives larger priors, more tokens or higher resolution, which is what matters in practice. Only rFID, gFID and AR loss are reported, without IS, precision/recall or multiple seeds, and on ImageNet the CFG schedule is tuned separately for each method (scale 20 vs. 24, App. F.2), which makes a single gFID difference harder to read. Finally, the only visual results are a few CIFAR-10 samples in App. G. There are no ImageNet generations and no reconstruction examples, which I would expect in any tokenizer paper.
-
The theory is mostly an adaptation of existing tools, so the paper's case rests on the experiments, and the experiments do not show where the gains come from. Theorem 3.1 is, as the paper itself says, a decomposition that follows from Bayes' rule. The aggregate prior-matching objective is in the line of WAE, AAE and InfoVAE, which the paper cites in Sec. 3.2. The Wasserstein-gradient-flow update has the same score-difference form as DMD and Diff-Instruct, as noted in Sec. 4.6, here applied to discrete tokens through a straight-through relaxation. Given this, I would expect careful ablations showing that the derived mechanism is what drives the improvement, but these are missing. The reference model in the update is a randomly initialised, frozen 4-layer AR Transformer with logits sharpened at temperature 0.1 (App. F.3), not the prior that will later model the tokens. At this temperature, a random network's next-token predictions are close to deterministic, so in practice the update pushes the tokenizer towards token sequences that a fixed random function can predict from their prefix, i.e. towards lower conditional token entropy. This could by itself explain why the Stage-II AR loss drops, and it is a different thing from the prior consistency that TVC and the derivation in Sec. 4 are about. The paper does not vary this choice (e.g. a trained reference, a different random seed, temperature 1.0), nor the matching weight λ or the proxy size, so it is unclear whether the gains come from prior matching as derived or from a simpler regularisation effect. Relatedly, App. C notes that "direct AR cross-entropy can decrease through lower token entropy as well as a closer fit", yet codebook usage and token entropy are not reported, so the large drop in AR loss on CIFAR-10 (0.78 to 0.42) may partly reflect lower entropy. The comparisons have similar issues. The rows in Tables 1 and 3 are labelled "TiTok", "FlowMo", "SelfTok" and so on, but they are the authors' re-implementations of individual techniques in a shared backbone without adversarial losses, not the published models, and in Table 1 the checkpoints are taken at very different iteration counts (50K–500K) to reach rFID ≈ 5. LARP and CRT, which address the same problem most directly, are discussed in App. C but not evaluated, and the direct AR-loss experiment in Table 4 uses a different setup from Tables 1–3, so it cannot stand in for that comparison.
Questions for authors
- How does wAR-Tok compare at system level with published AR tokenizers on ImageNet 256×256, and does the gain hold for larger generation priors and more tokens per image?
- How do the results change with a different reference model (a trained AR reference, a different random initialisation, temperature 1.0), with the prior-matching weight λ, and with the proxy size?
- Please report codebook usage and the entropy of the token distribution for wAR-Tok and the baseline. Is the drop in AR loss explained by lower token entropy?
- How does wAR-Tok compare with LARP and CRT on the same setup as Tables 1–3?
- For ImageNet, what are the gFID values under a shared CFG schedule and over several seeds? Please also include ImageNet generations and reconstruction examples.
3 comments
What is measured on ImageNet, at what scale and under which CFG schedules?
reviewer (q1+q5): How does wAR-Tok compare at system level with published AR tokenizers on ImageNet 256×256, and does the gain hold for larger generation priors and more tokens per image?
For ImageNet, what are the gFID values under a shared CFG schedule and over several seeds? Please also include ImageNet generations and reconstruction examples.
Table 2 reports one gFID per method under its own tuned CFG schedule: 6.32 for the baseline and 5.42 for wAR-Tok. The separately trained prior’s evaluation loss falls from 7.89 to 7.07 before sampling, so this prediction gain is independent of CFG. The visual comparisons in Appendix G are on CIFAR-10.
We observe gains with a 50M prior on CIFAR-10 and a 115M prior on ImageNet. Each comparison fixes the prior within its dataset, so these results do not establish a trend with prior size or sequence length.
How does wAR-Tok compare with published systems and the closest alignment methods?
reviewer (q1+q4): How does wAR-Tok compare at system level with published AR tokenizers on ImageNet 256×256, and does the gain hold for larger generation priors and more tokens per image?
How does wAR-Tok compare with LARP and CRT on the same setup as Tables 1–3?
TiTok-S-128 and CRT-AR-111M both report lower gFID than wAR-Tok on ImageNet 256×256. We believe this gap is due in part to differences in training compute, codebook size, and model architecture.
Thanks for the detailed reply. I like the theoretical framing and the derivations in the paper. At the same time, since the theory builds on existing tools, I feel the paper still needs fuller experimental validation to make the whole story feel complete, and also a fair expectation within the scope of this kind of work. Since no new experimental results were added in the response, I will keep my score at 2. Thanks again for all your efforts!!
Thank you for your detailed review. We appreciate your recognition of the simplicity of our method and the consistent gains in our controlled comparisons. We address your questions and concerns below.
What do the CIFAR-10 controls compare?
The rows in Tables 1 and 3 identify training techniques implemented on a common CIFAR-10 backbone. Table 1 selects checkpoints at comparable reconstruction quality, so step counts differ. At rFID near 5, wAR-Tok improves gFID from 17.60 to 14.50 relative to tail dropout. Table 3 holds tokenizer training at 400K steps. Here wAR-Tok achieves gFID 11.30 versus 22.76 for the best alternative, and it improves reconstruction as well as generation over tail dropout.
What does the method add to earlier distribution-matching work?
Theorem 3.1 starts from Bayes’ rule. Its design implication follows from Corollary 3.2, Case 1. If the decoder matches the data-anchored conditional distribution of images given tokens and the prior matches the aggregate token distribution, their product matches the joint and posterior consistency follows. This explains why aggregate matching suffices for prior consistency in WAE, AAE and InfoVAE (Appendix D.3). It also motivates Eq. (2): the tokenizer retains its reconstruction objective and receives the aggregate prior signal while the encoding is being learned.
Starting from Wasserstein gradient flow, we derive the prior-matching term ∇zlogq−∇zlogpref in Eq. (3), recovering the continuous score-difference form shared by the distribution-matching gradients of DMD and Diff-Instruct. For hard tokens, Eq. (4) compares the proxy and reference conditional log probabilities at each prefix. Two AR forward predictions supply the candidate costs, and Prob-STE passes the local signal to the encoder.
How do the reference, matching weight and proxy enter training?
Fitting a prior to existing tokens teaches it the dependencies of an encoding already chosen. Our frozen causal reference gives the tokenizer a fixed prediction target while it learns the encoding. Changing the initialization or using a trained reference changes that target. A temperature of 0.1 sharpens its preferences, with the degree of concentration depending on the logit gaps.
The matching weight sets the strength of alignment relative to reconstruction. The proxy estimates the current token distribution. The reported results use weight 1 and proxies of 17M on CIFAR-10 and 40M on ImageNet (Appendix F.3).
What does the lower Stage-II loss tell us?
The prior term in Eq. (2) is the cross-entropy to the reference minus the entropy of the aggregate token distribution. At a fixed prefix, swapping two unequal candidate probabilities preserves conditional entropy but changes the conditional KL if the reference gives them different probabilities. Our objective can therefore distinguish encodings that entropy alone scores identically.
The Stage-II loss depends on both conditional entropy and the fit of the new generation prior (Appendix C.2). Its decrease alone does not tell us how many codes were used or how much entropy changed. Table 1 also shows better image generation at comparable reconstruction quality.