acceptodds
Under review as a conference paper at ICLR 2027

Learning Discrete Autoregressive Priors with Wasserstein Gradient Flow

Abstract

Discrete image generation commonly uses two-stage training: a tokenizer is first trained for reconstruction, and a prior is then fitted to the tokens from the frozen tokenizer. This decoupling leaves the tokenizer unaware of the model that will later generate its tokens. As a result, the learned tokens may preserve image information well but still be difficult for an autoregressive (AR) prior to predict from left to right. We analyze this mismatch using Tripartite Variational Consistency (TVC), which decomposes latent-variable learning into three consistency conditions: conditional-likelihood consistency, prior consistency, and posterior consistency. TVC shows that two-stage training preserves the reconstruction side but leaves prior consistency outside the tokenizer objective: the overall token distribution is fixed before the AR prior participates in training. Motivated by this view, we add a distribution-level prior-matching signal during tokenizer training, while keeping the reconstruction objective unchanged. We optimize this signal with a Wasserstein-gradient-flow update. For hard categorical tokens, we approximate the update with a token-level contrast between an auxiliary AR model that tracks the tokenizer's current token distribution and a frozen AR reference. Computing this tokenizer update requires only forward passes through the two AR models; the proxy uses its own cross-entropy loss to track the current tokens. After tokenizer training, a separate AR prior is trained for generation. The resulting tokenizer, wAR-Tok, reduces AR loss and improves generation FID on CIFAR-10 and ImageNet at comparable reconstruction quality.

Primary area:
generative models
Keywords:
Image Tokenization, Autoregressive Image Generation, Discrete Representation Learning, Distribution Matching, Wasserstein Gradient Flow

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KtdxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between discrete image tokenizers trained only for reconstruction and the autoregressive (AR) priors later fitted to their tokens. It introduces Tripartite Variational Consistency (TVC), a view of latent-variable learning that separates likelihood, prior and posterior consistency, and uses it to argue that standard two-stage training leaves prior consistency outside the tokenizer objective. To address this, the authors add a distribution-level prior-matching term to tokenizer training and optimise it with a Wasserstein-gradient-flow update, which for discrete tokens is approximated by a per-token contrast between an online AR proxy and a frozen AR reference. The resulting tokenizer, wAR-Tok, is evaluated on CIFAR-10 and ImageNet 256×256, where it lowers AR loss and generation FID at comparable reconstruction quality.

Critical strengths
  1. The paper works on a relevant problem. Making the tokenizer aware of the prior that will later model its tokens is an open question for AR image generation, and the TVC view states clearly what two-stage training leaves out.
  2. The final method is simple and cheap. It only needs forward passes through two small AR models and adds about 10% to tokenizer training cost (App. F.3), so it would be easy to adopt if the gains hold up.
  3. Within the authors' controlled setup, the improvements are consistent. AR loss and gFID go down under both the matched-rFID (Table 1) and matched-iteration (Table 3) protocols, and the ImageNet comparison points in the same direction.
Critical weaknesses
  1. My main concern is that the experimental evaluation is far too limited for a tokenizer paper. Sections 2–4, about five pages, are spent on theory and derivation, while the experiments take up roughly half a page. Most results are on CIFAR-10 at 32×32 with 62 tokens per image, where the best gFID is still 11.3–14.5. The ImageNet 256×256 evidence (Table 2) is a single pair of rows, the authors' own 1D-tokenizer baseline and wAR-Tok: gFID improves from 6.32 to 5.42, but rFID gets worse, from 2.05 to 2.35. There is no system-level comparison with published AR tokenizers on ImageNet. This is a notable gap, because the works this paper builds on and compares against (e.g. TiTok, FlexTok, LARP) were themselves evaluated at ImageNet scale, or on standard video benchmarks in the case of LARP, with system-level comparisons; TiTok, for instance, reports a gFID of 1.97 on ImageNet 256×256, far below the 5.42 reported here. Only one tokenizer size and one prior size are used per dataset, so we do not know whether the gain survives larger priors, more tokens or higher resolution, which is what matters in practice. Only rFID, gFID and AR loss are reported, without IS, precision/recall or multiple seeds, and on ImageNet the CFG schedule is tuned separately for each method (scale 20 vs. 24, App. F.2), which makes a single gFID difference harder to read. Finally, the only visual results are a few CIFAR-10 samples in App. G. There are no ImageNet generations and no reconstruction examples, which I would expect in any tokenizer paper.

  2. The theory is mostly an adaptation of existing tools, so the paper's case rests on the experiments, and the experiments do not show where the gains come from. Theorem 3.1 is, as the paper itself says, a decomposition that follows from Bayes' rule. The aggregate prior-matching objective is in the line of WAE, AAE and InfoVAE, which the paper cites in Sec. 3.2. The Wasserstein-gradient-flow update has the same score-difference form as DMD and Diff-Instruct, as noted in Sec. 4.6, here applied to discrete tokens through a straight-through relaxation. Given this, I would expect careful ablations showing that the derived mechanism is what drives the improvement, but these are missing. The reference model pθp_\theta in the update is a randomly initialised, frozen 4-layer AR Transformer with logits sharpened at temperature 0.1 (App. F.3), not the prior that will later model the tokens. At this temperature, a random network's next-token predictions are close to deterministic, so in practice the update pushes the tokenizer towards token sequences that a fixed random function can predict from their prefix, i.e. towards lower conditional token entropy. This could by itself explain why the Stage-II AR loss drops, and it is a different thing from the prior consistency that TVC and the derivation in Sec. 4 are about. The paper does not vary this choice (e.g. a trained reference, a different random seed, temperature 1.0), nor the matching weight λ or the proxy size, so it is unclear whether the gains come from prior matching as derived or from a simpler regularisation effect. Relatedly, App. C notes that "direct AR cross-entropy can decrease through lower token entropy as well as a closer fit", yet codebook usage and token entropy are not reported, so the large drop in AR loss on CIFAR-10 (0.78 to 0.42) may partly reflect lower entropy. The comparisons have similar issues. The rows in Tables 1 and 3 are labelled "TiTok", "FlowMo", "SelfTok" and so on, but they are the authors' re-implementations of individual techniques in a shared backbone without adversarial losses, not the published models, and in Table 1 the checkpoints are taken at very different iteration counts (50K–500K) to reach rFID ≈ 5. LARP and CRT, which address the same problem most directly, are discussed in App. C but not evaluated, and the direct AR-loss experiment in Table 4 uses a different setup from Tables 1–3, so it cannot stand in for that comparison.

Questions for authors
  1. How does wAR-Tok compare at system level with published AR tokenizers on ImageNet 256×256, and does the gain hold for larger generation priors and more tokens per image?
  2. How do the results change with a different reference model (a trained AR reference, a different random initialisation, temperature 1.0), with the prior-matching weight λ, and with the proxy size?
  3. Please report codebook usage and the entropy of the token distribution for wAR-Tok and the baseline. Is the drop in AR loss explained by lower token entropy?
  4. How does wAR-Tok compare with LARP and CRT on the same setup as Tables 1–3?
  5. For ImageNet, what are the gFID values under a shared CFG schedule and over several seeds? Please also include ImageNet generations and reconstruction examples.
3 comments

Thank you for your detailed review. We appreciate your recognition of the simplicity of our method and the consistent gains in our controlled comparisons. We address your questions and concerns below.

What do the CIFAR-10 controls compare?

reviewer (w1+w2): Sections 2–4, about five pages, are spent on theory and derivation, while the experiments take up roughly half a page. Most results are on CIFAR-10 at 32×32 with 62 tokens per image, where the best gFID is still 11.3–14.5.

The rows in Tables 1 and 3 are labelled "TiTok", "FlowMo", "SelfTok" and so on, but they are the authors' re-implementations of individual techniques in a shared backbone without adversarial losses, not the published models, and in Table 1 the checkpoints are taken at very different iteration counts (50K–500K) to reach rFID ≈ 5.

The rows in Tables 1 and 3 identify training techniques implemented on a common CIFAR-10 backbone. Table 1 selects checkpoints at comparable reconstruction quality, so step counts differ. At rFID near 5, wAR-Tok improves gFID from 17.60 to 14.50 relative to tail dropout. Table 3 holds tokenizer training at 400K steps. Here wAR-Tok achieves gFID 11.30 versus 22.76 for the best alternative, and it improves reconstruction as well as generation over tail dropout.

What does the method add to earlier distribution-matching work?

reviewer (w2): The theory is mostly an adaptation of existing tools, so the paper's case rests on the experiments, and the experiments do not show where the gains come from. Theorem 3.1 is, as the paper itself says, a decomposition that follows from Bayes' rule.

The aggregate prior-matching objective is in the line of WAE, AAE and InfoVAE, which the paper cites in Sec. 3.2. The Wasserstein-gradient-flow update has the same score-difference form as DMD and Diff-Instruct, as noted in Sec. 4.6, here applied to discrete tokens through a straight-through relaxation.

Theorem 3.1 starts from Bayes’ rule. Its design implication follows from Corollary 3.2, Case 1. If the decoder matches the data-anchored conditional distribution of images given tokens and the prior matches the aggregate token distribution, their product matches the joint and posterior consistency follows. This explains why aggregate matching suffices for prior consistency in WAE, AAE and InfoVAE (Appendix D.3). It also motivates Eq. (2): the tokenizer retains its reconstruction objective and receives the aggregate prior signal while the encoding is being learned.

Starting from Wasserstein gradient flow, we derive the prior-matching term ∇zlog⁡q−∇zlog⁡pref\nabla_z\log q-\nabla_z\log p_{\rm ref} in Eq. (3), recovering the continuous score-difference form shared by the distribution-matching gradients of DMD and Diff-Instruct. For hard tokens, Eq. (4) compares the proxy and reference conditional log probabilities at each prefix. Two AR forward predictions supply the candidate costs, and Prob-STE passes the local signal to the encoder.

How do the reference, matching weight and proxy enter training?

reviewer (q2): How do the results change with a different reference model (a trained AR reference, a different random initialisation, temperature 1.0), with the prior-matching weight λ, and with the proxy size?

Fitting a prior to existing tokens teaches it the dependencies of an encoding already chosen. Our frozen causal reference gives the tokenizer a fixed prediction target while it learns the encoding. Changing the initialization or using a trained reference changes that target. A temperature of 0.1 sharpens its preferences, with the degree of concentration depending on the logit gaps.

The matching weight sets the strength of alignment relative to reconstruction. The proxy estimates the current token distribution. The reported results use weight 1 and proxies of 17M on CIFAR-10 and 40M on ImageNet (Appendix F.3).

What does the lower Stage-II loss tell us?

reviewer (q3): Please report codebook usage and the entropy of the token distribution for wAR-Tok and the baseline. Is the drop in AR loss explained by lower token entropy?

The prior term in Eq. (2) is the cross-entropy to the reference minus the entropy of the aggregate token distribution. At a fixed prefix, swapping two unequal candidate probabilities preserves conditional entropy but changes the conditional KL if the reference gives them different probabilities. Our objective can therefore distinguish encodings that entropy alone scores identically.

The Stage-II loss depends on both conditional entropy and the fit of the new generation prior (Appendix C.2). Its decrease alone does not tell us how many codes were used or how much entropy changed. Table 1 also shows better image generation at comparable reconstruction quality.

What is measured on ImageNet, at what scale and under which CFG schedules?

reviewer (q1+q5): How does wAR-Tok compare at system level with published AR tokenizers on ImageNet 256×256, and does the gain hold for larger generation priors and more tokens per image?

For ImageNet, what are the gFID values under a shared CFG schedule and over several seeds? Please also include ImageNet generations and reconstruction examples.

Table 2 reports one gFID per method under its own tuned CFG schedule: 6.32 for the baseline and 5.42 for wAR-Tok. The separately trained prior’s evaluation loss falls from 7.89 to 7.07 before sampling, so this prediction gain is independent of CFG. The visual comparisons in Appendix G are on CIFAR-10.

We observe gains with a 50M prior on CIFAR-10 and a 115M prior on ImageNet. Each comparison fixes the prior within its dataset, so these results do not establish a trend with prior size or sequence length.

How does wAR-Tok compare with published systems and the closest alignment methods?

reviewer (q1+q4): How does wAR-Tok compare at system level with published AR tokenizers on ImageNet 256×256, and does the gain hold for larger generation priors and more tokens per image?

How does wAR-Tok compare with LARP and CRT on the same setup as Tables 1–3?

TiTok-S-128 and CRT-AR-111M both report lower gFID than wAR-Tok on ImageNet 256×256. We believe this gap is due in part to differences in training compute, codebook size, and model architecture.

Thanks for the detailed reply. I like the theoretical framing and the derivations in the paper. At the same time, since the theory builds on existing tools, I feel the paper still needs fuller experimental validation to make the whole story feel complete, and also a fair expectation within the scope of this kind of work. Since no new experimental results were added in the response, I will keep my score at 2. Thanks again for all your efforts!!

Reviewer_deduOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mismatch between reconstruction-trained discrete tokenizers and downstream autoregressive priors. It proposes TVC as a conceptual view of latent-variable training and WGF-DPD, addding distribution-level prior matching during tokenizer training. The experiments show clear improvements in AR loss and generation FID on CIFAR-10 and ImageNet.

Critical strengths
  1. The paper targets a real and important mismatch between tokenizer reconstruction and AR modeling. This is a meaningful problem, and the motivation for incorporating prior modeling into tokenizer training is clear.

  2. The method shows consistent gains in generation quality and AR loss. Improvements appear on both CIFAR-10 and ImageNet, including lower gFID and AR loss under controlled settings.

Critical weaknesses
  1. The discrete update relies on a relaxed token representation and Probability-STE, so the connection to exact Wasserstein gradient flow is only approximate.

  2. The direct AR-loss baseline uses a different setup, and the ImageNet study compares only against the basic 1D tokenizer rather than stronger alignment methods.

Questions for authors
  1. What happens if the frozen AR reference is changed across random seeds, model sizes, temperatures, or architectures? The current reference uses temperature 0.1, making this design choice especially important.

  2. Can the authors provide a fully matched comparison against direct AR-loss training using the same tokenizer, prior, training schedule, and compute?

  3. How sensitive are the results to the proxy quality? The paper notes that the proxy only approximately tracks the changing token distribution, but does not provide a clear ablation of this approximation error.

2 comments

Thank you for your thoughtful review. We appreciate your recognition of the improvements in generation quality and AR loss on both CIFAR-10 and ImageNet. We address your questions and concerns below.

What justifies the discrete approximation?

reviewer (w1): The discrete update relies on a relaxed token representation and Probability-STE, so the connection to exact Wasserstein gradient flow is only approximate.

Equation (3) gives the continuous Wasserstein gradient of the KL prior term. For hard tokens, the sequence KL still decomposes into prefix-conditional KL terms. At an observed prefix, let uu and rr be the proxy and reference probabilities. The directional derivative of DKL(u∥r)D_{\rm KL}(u\|r) for moving mass from ii to jj is log⁡(uj/rj)−log⁡(ui/ri)\log(u_j/r_j)-\log(u_i/r_i), exactly the difference in local alignment cost between those candidates. Equation (4) uses this comparison with the AR predictions held fixed and passes the signal through Prob-STE (§4.5).

How do the reference settings enter the update?

reviewer (q1): What happens if the frozen AR reference is changed across random seeds, model sizes, temperatures, or architectures? The current reference uses temperature 0.1, making this design choice especially important.

Changing the seed, size or architecture changes the reference and can change the target for a given prefix. At fixed logits, lowering the temperature from 1 to 0.1 multiplies pairwise log odds by ten. How concentrated the distribution becomes depends on the original logit gaps. Tables 1–3 evaluate the frozen four-layer reference at temperature 0.1 with the architecture specified for each dataset in Appendix F.3.

What does the direct AR-loss comparison test?

reviewer (q2): Can the authors provide a fully matched comparison against direct AR-loss training using the same tokenizer, prior, training schedule, and compute?

Table 4 tests how the direct AR-loss weight affects the reconstruction–generation trade-off in a separate 1D setup. Increasing the weight from 0 to 0.3 improves gFID from 19.32 to 14.45 with little change in reconstruction. At weight 3, generation and reconstruction both deteriorate sharply (gFID 61.4, rFID 18.8).

At a given prefix, direct AR supervision assigns a lower local cost to a more probable token under the guiding prior. WGF-DPD also accounts for how much probability the current tokenizer already assigns that candidate relative to the reference.

What does the ImageNet comparison establish?

reviewer (w2): the ImageNet study compares only against the basic 1D tokenizer rather than stronger alignment methods.

The CIFAR-10 experiments provide a self-contained comparison of tokenizer training methods at matched reconstruction quality and at matched training steps (Tables 1 and 3). Repeating the full comparison on ImageNet was beyond our compute budget. On ImageNet, adding WGF-DPD to the basic 1D tokenizer improved generation quality, reducing gFID from 6.32 to 5.42 under the respective tuned CFG schedules (Table 2).

How does proxy error enter the update?

reviewer (q3): How sensitive are the results to the proxy quality? The paper notes that the proxy only approximately tracks the changing token distribution, but does not provide a clear ablation of this approximation error.

For a fixed tokenizer distribution qq, the true sequence KL minus the expected proxy–reference log ratio is exactly DKL(q∥qψ)D_{\rm KL}(q\|q_\psi) (§4.6). At a given prefix, the proxy’s log-probability error for a candidate enters its local cost directly. We fit the proxy to freshly produced tokens at every step to track the changing distribution.

Thank you for the detailed response and further clarification. However, my concern about insufficient experimental comparison on ImageNet remains unresolved. Furthermore, I agree with reviewer Ktdx's comment that the theoretical framework of this paper is primarily built upon existing theoretical tools and the derivation from theoretical analysis to practical discrete updates involves multiple assumptions and approximations, requiring more thorough experimental verification. Therefore, I tend to maintain my original rating.

Reviewer_bssJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the two-stage pipeline used for autoregressive (AR) image generation. In Stage I a tokenizer is trained, and in Stage II an AR model is trained on the tokenizer's tokens. The two stages are usually trained separately, so the tokenizer does not know that an AR model will later generate its tokens. This creates a gap between the two stages: a tokenizer with good reconstruction quality does not always give good generation quality in Stage II.

To address this gap, the authors propose a theoretical framework, Tripartite Variational Consistency (TVC), and use it to explain both ELBO training and two-stage training. They then add a KL term between the aggregate posterior and an AR reference prior to tokenizer training, which they call the Distributional Prior Divergence (DPD). To optimize this term, the authors:

  • derive a score-difference update from Wasserstein gradient flow (WGF);
  • use an online AR proxy to estimate the score of the aggregate posterior;
  • pass the signal back to the encoder with Prob-STE.

Experiments on CIFAR-10 and ImageNet 256×256 show that AR generation quality improves at similar reconstruction quality.

Critical strengths
  1. The proposed TVC framework explains ELBO training and two-stage training from one theoretical view. The reasoning is clear and consistent.
  2. The DPD objective connects tokenizer training with AR model training. The tokenizer can learn latent tokens that are easier for the AR model to generate, instead of being trained only for reconstruction.
Critical weaknesses
  1. The step from continuous WGF to hard discrete tokens is not valid

Continuous WGF works with distributions that have a continuous density, and it uses the spatial gradient

∇zlog⁡q(z).\nabla_z \log q(z).

A hard categorical distribution is a set of Dirac masses, so it has no normal Euclidean score of the kind used in the paper. A discrete Wasserstein gradient flow would also need:

  • a ground metric between token states;
  • a graph structure or transition edges;
  • a discrete continuity equation;
  • a transport flux.

The paper simply extends the one-hot variable to continuous values and obtains ∇zlog⁡P=log⁡π\nabla_z \log P = \log \pi. This is a usable surrogate, but it is not a true discrete Wasserstein gradient.

  1. Eq. (4) is not the full gradient of the AR joint probability

The AR joint log-probability is

log⁡p(z)=∑tzt⊤at(z<t),\log p(z) = \sum_t z_t^\top a_t(z_{<t}),

where ata_t is the log-probability vector at position tt. The gradient with respect to an earlier token zsz_s is

= a_s(z_{<s}) + \sum_{t>s} \left( \frac{\partial a_t(z_{<t})}{\partial z_s} \right)^\top z_t .$$ The paper keeps only the first term. It ignores how changing an early token changes all later conditional probabilities. So the method cannot both avoid backpropagating through the AR model and compute the score of the AR joint distribution. What the paper computes is a detached, teacher-forcing local signal, not the full $\nabla_z \log p(z)$. 3. The roles of the AR models are not clearly separated The paper uses $p_\theta$ for two different models: - the randomly initialized, frozen reference in Stage I (Sec. 4.2, Appendix F.3); - the separately trained generation prior in Stage II. The appendix also calls the proxy $q_\psi$ "joint" (Table 6, Appendix F.2). Because of this, it is easy to misread which models are updated during tokenizer training. For example, it is easy to misread the 300-epoch AR training on ImageNet as part of Stage I, when it is the Stage II prior. Please state clearly, for both CIFAR-10 and ImageNet: - whether the reference prior is updated during tokenizer training; - whether the reference prior is the same model as the final generation prior; - which AR outputs are detached; - when each AR model is trained; - whether CIFAR-10 and ImageNet use the same algorithm and the same reference setting (size, temperature). Please also use different symbols for the reference prior and the generation prior. 4. There is no theoretical basis for a fixed, random, low-temperature reference If the reference is a fixed random Transformer, it has learned nothing about the data. A low temperature only makes its random logits and random context dependence stronger. The gains may therefore come from other causes: - random token preferences; - lower token entropy; - lower codebook usage; - collapse of sequence patterns; - general random regularization; - a different trade-off between reconstruction and generation. The work does not show that the causal mask of a randomly initialized AR model is enough to give a useful AR inductive bias.
Questions for authors

Which are presented in the weakness; please refer to the above.

1 comment

Thank you for your careful review. We appreciate your recognition that TVC gives a unified view of ELBO training and two-stage training. We address your questions and concerns below.

What does the WGF-inspired update retain at a fixed prefix?

reviewer (w1+w2) : This is a usable surrogate, but it is not a true discrete Wasserstein gradient.

The paper keeps only the first term. It ignores how changing an early token changes all later conditional probabilities.

Equation (3) gives the Wasserstein particle gradient of the KL prior term for smooth positive densities. For hard tokens, §4.5 uses the linear extension z⊤log⁡pz^\top\log p. It equals the categorical log probability at each one-hot candidate and has gradient log⁡p\log p with respect to the relaxed token. Both AR models read the full prefix. We hold their predictions fixed during tokenizer alignment, which omits derivatives through later predictions.

The KL chain rule gives this local update a discrete interpretation. At a fixed prefix, let uu and rr be the proxy and reference distributions and let gk=log⁡(uk/rk)g_k=\log(u_k/r_k). Moving probability mass from ii to jj at uu changes the conditional KL at rate gj−gig_j-g_i. Switching between those candidates changes Algorithm 1’s local cost by exactly gj−gig_j-g_i. Prob-STE passes this signal to the tokenizer.

How are the three AR models trained?

reviewer (w3) : The roles of the AR models are not clearly separated

  • whether the reference prior is updated during tokenizer training;
  • whether the reference prior is the same model as the final generation prior;
  • which AR outputs are detached;
  • when each AR model is trained;
  • whether CIFAR-10 and ImageNet use the same algorithm and the same reference setting (size, temperature).

Please also use different symbols for the reference prior and the generation prior.

Both datasets use the same algorithm. In Stage I, the reference prefp_{\rm ref} is a randomly initialized four-layer causal AR model, frozen at temperature 0.1. It has the same architecture as the proxy qψq_\psi on each dataset (about 17M parameters on CIFAR-10 and 40M on ImageNet).

At each step, the proxy trains on detached tokens. For the tokenizer update, we detach both AR predictions and pass their log-probability difference through Prob-STE. This is what “joint” means in Table 6: the proxy trains alongside the tokenizer with its own loss. In Stage II we freeze the tokenizer and train a new generation prior pgenp_{\rm gen}. The 300 epochs on ImageNet refer to this generation prior (Algorithm 1, Appendix F).

Why use a frozen, randomly initialized AR reference?

reviewer (w4) : There is no theoretical basis for a fixed, random, low-temperature reference

The gains may therefore come from other causes:

  • random token preferences;
  • lower token entropy;
  • lower codebook usage;
  • collapse of sequence patterns;
  • general random regularization;
  • a different trade-off between reconstruction and generation.

The work does not show that the causal mask of a randomly initialized AR model is enough to give a useful AR inductive bias.

Two encodings can reconstruct the same images equally well yet differ in how easily an AR model can predict their tokens. Fitting a prior to the current tokens asks it to learn dependencies the tokenizer has already chosen. We want left-to-right predictability to influence the choice of encoding itself.

The proxy tracks the current tokens. The frozen causal reference gives a fixed next-token target whenever the same prefix occurs (§4.2). If the tokenizer matched those conditionals exactly, the four-layer reference could represent the entire sequence distribution. We test whether this modeling preference transfers by training a fresh generation prior.

Sharpening the reference does not by itself determine which token the update favors. Even the reference’s favorite token can be overrepresented by the proxy. Moving mass from that token to an underrepresented candidate then lowers the conditional KL to first order. On CIFAR-10, the separately trained prior has lower AR loss at comparable rFID (Table 1). At equal training steps, wAR-Tok also improves both generation and reconstruction over tail dropout (Table 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.