acceptodds
Under review as a conference paper at ICLR 2027

RAE-VAR: Coarse-to-Fine Generation as Constrained Allocation in Representation Space

Abstract

Representation autoencoders provide semantic features for image generation, but how should these features be organized for coarse-to-fine prediction? Much of their decodable content can be retained in a low-rank channel subspace, yet placing that information in the earliest tokens can make those tokens difficult to predict. We introduce RAE-VAR, guided by constrained allocation: a design principle, rather than an explicitly optimized objective, for making early scales informative without asking them to predict too much from a short context. Its staircase tokenizer grows spatial resolution and channel rank together, using a nested PCA basis and Fourier resampling while leaving the VAR transformer unchanged. On ImageNet , the 310M-parameter, 16-layer model reaches gFID in 43 epochs, surpassing released VAR's after 200 epochs with fewer generator epochs. At 200 epochs, RAE-VAR improves to with nine next-scale sampling steps and 655 tokens. Larger 20- and 24-layer models approach their released VAR counterparts with roughly one fifth of the generator training epochs. Controlled ablations support both the structured basis and Fourier resampling, beyond what final reconstruction quality alone explains. Linear probes further show that early prefixes contain class information before their decoded images become recognizable.

Primary area:
generative models
Keywords:
Visual Autoregressive Model, Representation AutoEncoder, Coarse-to-Fine Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r2ZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper replaces the VQGAN tokenizer of VAR with a DINOv2-based RAE and asks what coarse-to-fine should mean in representation space. Channel rank truncation preserves decodable content much better than spatial downsampling, but putting all that content into the first scale makes it hard to predict from short context. From this observation the paper proposes a staircase tokenizer that grows spatial resolution and channel rank together (nested PCA basis + Fourier resampling). At d16 it surpasses the official VAR-d16 with about one-fifth of the generator epochs, and comes close at d20 and d24. The idea is natural, but the key evidence is scattered over the appendix (App. A.3, C.1, C.4, etc.), and the fact that the headline basis spans only 8 classes is disclosed only in App. A.3. So from the paper alone it is hard to tell whether the gain comes from the proposed allocation principle.

Critical strengths
  1. The practicality is what strengthens this paper. The backbone is unchanged, the tokenizer size is similar (109M vs. 111M), the released VAR checkpoints are re-evaluated under one pipeline, and the sampler setting is fixed per family. Under these conditions the method outperforms the official VAR-d16 with about one-fifth of the generator epochs. This is the result that makes the paper worth considering.

  2. The "easy to decode is not easy to predict" insight. The rank-only schedule reconstructs well but collapses in generation, and the oracle first rank band restores it, as shown in App. C.4. This diagnosis is convincing and informative for next-scale tokenizer design in general, yet it would have been better if it were part of the main text.

Critical weaknesses
  1. One critical issue would be the fact that the headline gain cannot be attributed to the proposed principle. Rank allocation itself is never isolated. Tab. 4 ablates only the basis and the resampling inside the staircase, and there is no RAE-spatial-VAR + Fourier resampling result. Since the pooled staircase alone is worse than spatial, the possibility that the whole gain comes from Fourier resampling cannot be excluded. More importantly, the headline model and the ablations use different bases. The headline model uses a 400-image PCA basis that spans only 8 classes, while the ablations use a class-uniform training-split PCA basis. With the class-uniform basis, the staircase versus spatial gap is substantial at ep8 (6.28 vs. 7.58) but nearly vanishes at ep16 (5.36 vs. 5.37). So the headline gap appears only with the 8-class basis. The authors state that they have no explanation for this (App. A.3, C.1), and the front-loading account in Sec. 4.5 points the other way. So it is unknown how much of the 2.98/2.58 in Tab. 2 is attributable to the design. The removal of the 5×55\times5 step and the choice of the rank schedule are also not justified.

  2. The other issue is that every result is from a single run, and the depth claim in Tab. 3 remains unreliable. App. B.5 reports a mid-training checkpoint fluctuation of 3.34 at epoch 55, 3.70 at 70, and 3.41 at 85 for d16, and the paper itself says this is far above the ±0.03\pm0.03 50k-FID noise floor. Tab. 3 then claims that d20 and d24 approach the released checkpoints with gaps of +0.03+0.03 and +0.16+0.16, which are well inside that fluctuation. The App. B.5 fluctuation occurs during learning-rate decay, so it is not the same as seed variance, but that is exactly the point. The paper has no per-seed statistics at all, so there is no way to tell whether +0.03+0.03 reflects proximity or noise. The ablations in Tab. 4 rest on single ep8/ep16 checkpoints without cooldown as well.

Questions for authors
  1. What happens when only Fourier resampling (and PCA rotation at full rank) is applied to RAE-spatial-VAR? Can you show the independent contribution of rank allocation?

  2. What are the 40+3 and 200-epoch results for the class-uniform training-split PCA staircase?

  3. How were the rank schedule and the removal of the 5×55\times5 step decided? Were other schedules tried, for example one that includes the 5×55\times5 step?

  4. Can you report the standard deviation over at least two more seeds for the Tab. 3 depth comparison and the ep16 ablation in Tab. 4, since they show only marginal differences?

8 comments
AuthorsOpenReview ↗

Response to Reviewer r2Zp (1/2)

We thank the reviewer for the two strengths named: the result under an unchanged backbone and a single pipeline, and the "easy to decode is not easy to predict" diagnosis. Following the reviewer, we move the rank-only failure and the oracle first-rank-band test (supplying ground-truth tokens for the first rank band during rollout) from App. C.4 into Sec. 3, and the basis estimate and the class-uniform result into the main text.

W1 / Q1 / Q2 / Q3 (attribution; bases; schedule). During the discussion period we ran the full-rank ablation variant requested in Q1 and extended every Table-4 in the paper variant through the cooldown. VAR-d16, 50k-sample gFID; the control variants use tokenizers trained for 105k optimizer steps; each family is evaluated at its fixed sampling setting:

rFIDep8ep16ep4040+340+3 (40 ep + 3-ep cooldown)
RAE-VAR (class-uniform basis)0.9636.285.363.393.00
full-rank control (no rank restriction)1.0886.445.004.363.19
pooled resampling instead of Fourier1.3947.996.124.793.84
random basis instead of PCA1.38710.376.185.814.02
RAE-spatial-VAR1.0947.585.374.183.88
VQGAN-VAR3.3914.408.665.854.88
  • Q1 (Fourier and a full-rank PCA rotation on the spatial tokenizer). The full-rank control reaches 3.19 (vs. 3.88 for RAE-spatial-VAR and 3.00 for RAE-VAR), showing that PCA + Fourier contributes 0.69 and progressive rank allocation contributes 0.19 (22% of the 0.88 gain, or 6% relative to the full-rank control), on VAR's spatial ladder (1×11\times 1 to 16×1616\times 16) omitting the 5×55\times 5 scale (655 tokens).
  • Is the gain all Fourier resampling? No—neither Fourier resampling alone (4.02 with a random basis) nor PCA alone (3.84 with pooling) is sufficient; both are required and together carry 0.69 of the 0.88 gain.
  • Different bases (Q2). The class-uniform model reaches 3.00 at 40+340+3 and 2.55 at 200 epochs (matching the submitted basis's 2.98 and 2.58), confirming that the 0.88 gain over RAE-spatial-VAR does not depend on the 8-class basis, and the epoch-16 near-tie was transient (3.39 vs 4.18 at epoch 40). In the revision the main d16 row uses the class-uniform basis, so the main model and the ablations share one basis.
  • Early convergence vs. front-loading. At the endpoints the two bases agree, so there is no conflict at convergence. The class-uniform basis converges more slowly through epoch 16; we have no verified explanation and will say so.
  • Q3 (5×55\times 5 step and rank schedule). The schedule was fixed once to keep total tokens below the baseline (655<680655 < 680) without gFID tuning, with four rank levels in steps of 64. We compared schedule families (rank-only, rank after a spatial warm-up, joint staircase), not individual schedules; a schedule with 5×55\times 5 was not tried. The full-rank control bounds the rank schedule's contribution (0.19).
  • Consequence. Together, these results show that all components of our coarse-to-fine representation hierarchy are necessary and complementary: RAE tokens (4.88→3.884.88 \rightarrow 3.88 at 43 epochs), the PCA projection with exact Fourier low-pass resampling (3.88→3.193.88 \rightarrow 3.19), and progressive channel-rank allocation (3.19→3.003.19 \rightarrow 3.00 while reducing tokens from 680 to 655).
AuthorsOpenReview ↗

Response to Reviewer r2Zp (2/2)

W2 / Q4 (single runs; Table 3 in the paper). Following standard practice in ImageNet 256×256256\times 256 generative modeling (e.g., VAR, LlamaGen, MAR, DiT, VFMTok, DINO-Tok), our results are single training runs; a single 200-epoch d16 run costs about 1,200 GPU-hours, making multi-seed runs across Tables 3 and 4 in the paper not feasible given our limited compute budget. We did two things instead: every comparison our claims use is now read at a post-decay endpoint (all Table-4 in the paper variants extended to 40+340+3, i.e., 40 training epochs plus a 3-epoch linear cooldown; W1), and we measured the spread at those points.

We agree with the reading of App. B.5 and measured it further. At the same sampling setting (cfg =1.2= 1.2, top-p=0.92p = 0.92), the class-uniform d16 run shows the same behaviour: 3.39, 3.61, 3.56 and 3.29 at epochs 40, 50, 70 and 100, next to 3.71, 3.34, 3.70, 3.41 and 3.14 at epochs 40, 55, 70, 85 and 100 for the submitted run. While pre-decay checkpoints (at 62–86%62\text{--}86\% of peak learning rate) fluctuate by 0.2–0.4, post-decay endpoints across two independently trained d16 runs agree within 0.02 at 40+340+3 and 0.03 at 200 epochs (with two 50k sample sets of one checkpoint differing by at most 0.06).

Consequently, (i) we now evaluate all Table-4 in the paper ablations at post-decay endpoints (40+340+3; gaps of 0.84, 1.02, and 0.19), (ii) we revise our Table-3 in the paper wording to "within 0.2 gFID of the released checkpoints with about 4.6×4.6\times fewer generator epochs" (explicitly stating that +0.03+0.03 at d20 is parity and +0.16+0.16 at d24 is a deficit), and (iii) we anchor our main claims on the large d16 margins (0.60 against released VAR-d16, 0.88 against RAE-spatial-VAR, and 1.88 against VQGAN-VAR). The d20 (50+450+4) and d24 (70+570+5) cooldown branches start at epochs 50 and 70, inside the fluctuating window; the d16 pair suggests the decay does not carry such deviations into the endpoint, but we have not tested this at d20 and d24.

Presentation. We moved the basis provenance, class-uniform full-budget results, and rank-only failure analysis (with the oracle first-rank-band diagnosis) directly into Sec. 3 of the main paper.

Thank you for the responses and the additional experiments. The class-uniform basis results show that the headline gain does not depend on the 8-class basis, and the revised wording for Tab. 3 now matches what the numbers support. I also appreciate the honest answer on how the rank schedule was decided. However, although the response states that the rank-only failure analysis was moved into Sec. 3, the revised paper still only refers to App. C.4 for it (it would also have been better if the authors had marked where the changes were made).

The full-rank control shows that rank allocation accounts for only 0.19 of the 0.88 gain, while most of it already comes from the PCA projection and Fourier resampling without any rank restriction. I am not arguing that the 0.19 improvement is only marginal, and I acknowledge that the revised Sec. 4.4 and the conclusion now state that the rank schedule has a smaller effect. However, the title, abstract, and introduction still present constrained allocation as the central idea, so the framing is only partly updated.

The proposed method may consequently improve the performance, and since most of my concerns on the experiments have been addressed, I raise my soundness to 2. However, from what I have been given, the main framing of the paper still does not match where the gain comes from, so I keep my initial rating of 2.

Reviewer_we2cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes RAE-VAR, a coarse-to-fine autoregressive image generation framework that replaces VAR’s pixel-oriented VQGAN tokenizer with a representation autoencoder (RAE) built on DINOv2 features. The main idea is that representation features preserve substantial decodable information in low-rank channel subspaces, but predicting this information at early scales can be difficult because little conditioning context is available. The main contribution is a staircase tokenizer that progressively increases both spatial resolution and channel rank, using PCA basis, residual vector quantization, and Fourier resampling. This follows that early scales should carry useful information, but no more than the generator can predict from a short context which the paper states as constrained allocation. The VAR transformer itself is left unchanged. On class-conditional ImageNet-256, RAE-VAR-d16 reaches gFID 2.98 after 43 epochs, outperforming the released VAR-d16 (3.15 after 200 epochs), and improves to 2.58 after 200 epochs. Deeper d20 and d24 models come close to the released VAR models of the same depth with about one fifth of the training epochs.

Critical strengths
  1. The results on generation quality (a gFID 2.98 on 43 epochs / 2.58 on 200 epochs compared to 3.15 on 200 epochs) are compelling. They demonstrate that improving the tokenizer only without increasing generator capacity can improve the autoregressive generation.
  2. The paper provides observation that low-rank channel subspaces preserve substantially more decodable information than spatial downsampling at comparable information budgets with further investigation on how semantic information is distributed across coarse-to-fine tokens through reconstruction analyses and linear probes. These findings provide useful insights into designing token hierarchies for representation-based generation, beyond simply improving quality.
Critical weaknesses
  1. All main results use a PCA basis computed from only 400 validation images, covering just 8 of the 1,000 ImageNet classes according to Appendix A.3. In §4.4, the authors investigate a class-uniform training-set basis covering all classes. Although this basis achieves comparable reconstruction quality, RAE-VAR reaches only 5.36 gFID at epoch 16, nearly identical to RAE-spatial-VAR without the staircase which gives 5.37 and worse than the original basis which gives 4.70. This suggests that the effectiveness of the staircase tokenizer is sensitive to the PCA basis despite comparable reconstruction quality. While the authors acknowledge this gap, the comparison is limited to epoch 16, leaving it unclear whether the improvement persists at longer training durations. In particular, the headline 43-epoch result (2.98 gFID compared to 3.88 for RAE-spatial-VAR) is not evaluated with the class-uniform basis, making the robustness of the reported improvement difficult to assess.
  2. Constrained allocation is presented as the main contribution; however, the ablations do not isolate the incremental contribution of progressive channel-rank restriction. This makes it difficult to determine whether the improvements are driven by the proposed allocation principle or primarily by the PCA basis and Fourier resampling. Together with the sensitivity to the PCA basis discussed above, this leaves uncertainty about the robustness of the proposed method and which design choices are responsible for its gains.
  3. The method illustration (Figure 2) could be more explanatory and clearer in its presentation. Additionally, the qualitative comparisons in Figure 9 could be enlarged to make fine details easier to inspect, especially since the figure is already in the appendix. Including RAE-spatial-VAR and the headline 43-epoch RAE-VAR results would also help readers better assess the improvements introduced by the method. Overall, while the generation results are promising, the sensitivity to the PCA basis and the lack of an ablation isolating progressive channel-rank restriction make it difficult to determine whether the reported gains are robust and attributable to the proposed constrained allocation principle. These concerns motivate my current weak rejection recommendation. Demonstrating consistent improvements with alternative PCA bases, together with an ablation isolating the contribution of channel-rank restriction, would substantially strengthen the paper and could change my assessment.
Questions for authors
  1. Could you report two or three different bases (8 or 16 epochs ablations) built from different sets of images to investigate if results depend on the specific 400 image basis? If results are stable across different bases, this could help with a plausible explanation.
  2. (Clarification) Was the choice of specific 400-image subset fixed before evaluating generation performance?
  3. Could you report an experiment setting where you tested RAE-VAR (same PCA basis, Fourier resampling and scales) but removed the channel limit, so every step uses all 256 channels? This would show whether the channel limit itself is a reason for the improvement.
  4. (Clarification) How was the staircase schedule selected? Were alternative spatial-resolution and channel-rank schedules evaluated, and how sensitive is generation quality to these choices?
  5. Additional (not affecting my score). It could be useful to further discuss how RAE-VAR relates to recent approaches that organize visual information progressively for generation. In particular, Next Visual Granularity Generation (NVG) [1] explores progressive visual granularity beyond spatial resolution, while SpectralAR [2] also investigates frequency-structured tokenization for coarse-to-fine autoregressive generation. Although these approaches differ in their representations and generative formulations, they appear closely related to the paper's motivation of designing token hierarchies that organize information according to the generation process. A brief discussion of these connections could help clarify what distinguishes the proposed constrained allocation principle from existing approaches to organizing information across generation stages. This suggestion does not affect my judgement. [1] Next Visual Granularity Generation [https://arxiv.org/abs/2508.12811] [2] SpectralAR: Spectral Autoregressive Visual Generation [https://arxiv.org/abs/2506.10962]
6 comments
AuthorsOpenReview ↗

Response to Reviewer we2c (1/2)

We thank the reviewer for stating what would change the assessment: consistent gains with a different basis, and an ablation that isolates the channel-rank restriction. We ran both during the discussion period.

W1 (sensitivity to the basis; the 43-epoch result with the class-uniform basis). The basis of the submitted model was estimated once from a small image set (the first 400 images of the sorted validation split, 8 classes; App. A.3) and fixed before any tokenizer or generator was trained. We have now trained the class-uniform model through the full schedule (same recipe, same fixed sampling setting (cfg =1.2= 1.2, top-p=0.92p = 0.92)):

VAR-d16, 50k-sample gFIDRAE-spatial-VARRAE-VAR, 400-image basisRAE-VAR, class-uniform basis
tokenizer rFID1.0940.9550.963
epoch 8 / 167.58 / 5.376.05 / 4.706.28 / 5.36
epoch 40 (before cooldown)4.183.713.39
40+340+3 (40 ep + 3-ep cooldown)3.882.983.00
epoch 200—2.582.55
  • The epoch-16 near-tie with RAE-spatial-VAR (5.36 vs 5.37) is transient: the class-uniform model pulls ahead to 3.39 vs 4.18 at epoch 40 and 3.00 vs 3.88 at 40+340+3. Mid-training checkpoints are read near the peak learning rate, where such ties also occur for the submitted model (epoch 25: 4.32 vs 4.34) before the gap re-opens.
  • Both bases reach the same endpoints (2.98 and 3.00 at 43 epochs; 2.58 and 2.55 at 200 epochs), confirming that the 0.88–0.90 gain over RAE-spatial-VAR does not depend on the basis estimate. Sampled at each model's own optimum, the class-uniform model is 0.20 lower (2.37 vs 2.56).
  • The class-uniform basis converges more slowly through epoch 16. Sec. 4.4 read that early gap as the control not reproducing the main result; we replace this with the endpoint comparison.
  • In the revision, the main d16 row uses the class-uniform basis so that the main model and the ablations share one basis; the 400-image model is reported alongside.

W2 / Q3 (isolating the channel-rank restriction). We ran this control (new): the same spatial ladder (1×11\times 1 to 16×1616\times 16 without 5×55\times 5), frozen PCA projection and Fourier resampling, with (almost) all channels at every step (ranks 253–256253\text{--}256, which keeps the four per-rank SimVQ codebooks for ranks 64, 128, 192, and 256 on the same steps; the omitted directions carry 0.41% of the variance). VAR-d16, 50k-sample gFID; the control variants use tokenizers trained for 105k optimizer steps; each family is evaluated at its fixed sampling setting:

rFIDep8ep16ep4040+340+3
RAE-VAR (class-uniform basis)0.9636.285.363.393.00
full-rank control (no rank restriction)1.0886.445.004.363.19
pooled resampling instead of Fourier1.3947.996.124.793.84
random basis instead of PCA1.38710.376.185.814.02
RAE-spatial-VAR1.0947.585.374.183.88
VQGAN-VAR3.3914.408.665.854.88

The rank restriction accounts for 0.19 of the 0.88 gain over RAE-spatial-VAR, while the PCA projection with Fourier resampling accounts for 0.69. All three design choices—representation tokens (4.88→3.884.88 \rightarrow 3.88), the PCA subspace with band-limited Fourier resampling (3.88→3.193.88 \rightarrow 3.19), and progressive channel-rank allocation (3.19→3.003.19 \rightarrow 3.00 with fewer tokens)—are necessary and work together, as removing any single component degrades generation quality back toward or below RAE-spatial-VAR (3.19 without rank restriction, 3.84 without Fourier resampling, and 4.02 without the PCA basis). We will include the full-rank control in Table 4 in the paper and clarify each component's contribution in the text.

W3 (figures). We will redraw Fig. 2 to clarify the pipeline and schedule, and enlarged the appendix sample grids to full resolution (two per row) with RAE-spatial-VAR and 43-epoch RAE-VAR added on identical classes and seeds.

Q1 (two or three bases). We have three bases spanning a wide range of quality, measured as retained training-set variance at r=256r = 256:

basisvarianceep8ep1640+340+3ep200
random orthonormal (105k-step tokenizer)66.7%10.376.184.02—
400 images, 8 classes77.0%6.054.702.982.58
class-uniform, training split88.0%6.285.363.002.55

Both PCA bases substantially outperform the random basis at every epoch and converge to virtually identical endpoints (2.98 vs 3.00 at 40+340+3; 2.58 vs 2.55 at 200 epochs), although they come from very different image sets. A further class-uniform draw would not differ in practice: bases from class-uniform training and validation samples agree at top-kk subspace overlap of 0.98.

AuthorsOpenReview ↗

Response to Reviewer we2c (2/2)

Q2 (was the subset fixed before evaluating generation). Yes—the 400-image set was fixed once before any staircase tokenizer or generator was trained and was never tuned or selected on generation results.

Q4 (how the schedule was selected). The spatial sides follow VAR's spatial resolution ladder (1×11\times 1 to 16×1616\times 16) omitting the 5×55\times 5 scale, which keeps the token count below the baseline (655<680655 < 680), so the comparison cannot be credited to more tokens or sampling steps. The ranks are four levels in steps of 64, each held for two steps, with full rank for the last three steps. The schedule was fixed once by a token-count rule (655<680655 < 680 tokens) rather than tuned on gFID, with the full-rank control directly bounding the rank schedule's 0.19 contribution. Fine-grained sensitivity to individual rank increments or the 5×55\times 5 step was not swept, as our goal was to validate a simple, untuned schedule.

Q5 (NVG, SpectralAR). NVG keeps the spatial resolution and varies the number of distinct tokens per stage, whereas SpectralAR orders tokens by spatial frequency. Our Fourier resampling is closely related to SpectralAR, as exact spectral low-pass coarse steps are one of the key components behind our gain (W2). Unlike NVG and SpectralAR, RAE-VAR operates in a semantic representation space, constructs band-limited low-pass views of its PCA subspace, and jointly allocates channel rank with spatial scale so that early steps remain predictable from a short context.

Thank you for the detailed response and additional analysis. Rebuttal resolves my concern about basis sensitivity. However, I still have concerns about how the framing of improvement. RAE-VAR row in the ablation table uses the full-budget tokenizer, while full-rank control uses a tokenizer trained for 105k steps. PCA reference is actually behind the full-rank control at epoch 16 (5.29 vs. 5.00) under fixed budget. I do not expect another analysis on this considering the remaining time, but this should be clearly stated alongside the reported 0.19 improvement. Even if 0.19 gap is accepted, the PCA projection with Fourier resampling accounts for roughly 78% of the total 0.88 improvement over RAE-spatial-VAR, leaving only about 22% attributable to progressive rank allocation. Based on these, I do not think the evidence supports constrained allocation as main component. There are also two related points: (i) The response points out to reduction from 680 to 655 tokens to rank allocation. However, full-rank control also uses the same 655-token ladder so it reads that this reduction comes from omitting the 5×5 scale not rank allocation. (ii) The statement that removing any single component brings quality “back toward or below RAE-spatial-VAR” does not hold for the full-rank control, which still performs better (3.19 vs. 3.88). If I am misinterpreting the results, I would appreciate a correction. Otherwise, I encourage the authors to revise the contribution and discussion to better reflect the main contributions.

Reviewer_p6tZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RAE-VAR, which uses Representation Autoencoder (RAE) features for VAR-style next-scale generation. Its staircase quantizer jointly increases spatial resolution and channel rank, using a PCA-ordered basis, SimVQ quantization, and Fourier resampling, while keeping the VAR-d16 generator unchanged. On ImageNet 256×256, the authors report a gFID of 2.98 after 43 generator epochs and 2.58 after 200 epochs, and analyze why rank-only schedules fail during autoregressive rollout.

Critical strengths
  • Clear motivation. The paper distinguishes features that are easy to decode from tokens that are easy to predict autoregressively, and uses this distinction to motivate joint allocation of spatial scale and channel rank.

  • Useful controlled comparison. With the same VAR-d16 backbone and evaluation pipeline, RAE-VAR’s 43-epoch result outperforms the authors’ re-evaluation of the official VAR-d16 checkpoint trained for 200 epochs. It also improves further at 200 epochs.

  • Specific ablations and failure analysis. The paper compares PCA and random bases, Fourier and pooled resampling, and analyzes why rank-only schedules fail in free-running generation. It also documents its experimental protocol and limitations in detail.

Critical weaknesses
  • The PCA basis used for the main results may not be representative. It is estimated from the first 400 images in the validation split, which cover only 8 ImageNet classes. The authors report that a class-uniform training-set PCA retains more variance, but do not retrain the full main model with that basis. In addition, the PCA-versus-random-basis ablation changes both the retained subspace and the channel ordering, so it does not isolate the effect of PCA ordering.

  • The final contribution of PCA and Fourier resampling remains uncertain. At epoch 16, the PCA reference improves gFID by about 0.89 and 0.83 relative to the random-basis and pooled-resampling variants, respectively. These are measurable gains, but the controlled ablations stop at epoch 16. There are no multiple-seed results or uncertainty estimates, and it is unclear whether these gains persist after full training. The effectiveness of a PCA basis under distribution shift is also untested.

  • The qualitative evidence is limited. The final samples are shown in small grids. They are generally recognizable, but the figures do not make a substantial visual quality improvement compelling. Experiments are also limited to class-conditional ImageNet 256×256, so transfer to other data domains remains untested.

Questions for authors
  1. Could the authors train the main model with a class-balanced PCA basis estimated from the training split, and report a same-budget comparison against the current 400-image basis, including results across multiple seeds?

  2. Do the PCA and Fourier gains persist after full generator training? Could longer-budget ablations and comparisons against other anti-aliased low-pass resampling methods help establish this?

  3. Can the authors provide a controlled generation comparison with closely related foundation-model tokenizers, particularly VFMTok[1] and DINO-Tok[2], on ImageNet 256×256 using a matched backbone, training budget, and sampling/evaluation protocol?

[1] Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation [2] DINO-Tok: Adapting DINO for Visual Tokenizers

3 comments
AuthorsOpenReview ↗

Response to Reviewer p6tZ (1/2)

We thank the reviewer for finding the decode-versus-predict distinction well motivated and the same-backbone comparison useful. Results marked new were obtained during the discussion period.

W1 / Q1 (basis; seeds). The basis of the submitted model was estimated once from a small image set (the first 400 images of the sorted validation split, 8 classes; App. A.3). It was fixed before any tokenizer or generator was trained and was not changed or selected on generation results. To test whether the results depend on this estimate, we trained the same tokenizer and generator with a basis from a class-uniform sample of the training split through the full schedule (new; same recipe, same fixed sampling setting (cfg =1.2= 1.2, top-p=0.92p = 0.92)):

VAR-d16, 50k-sample gFIDRAE-spatial-VARRAE-VAR, 400-image basisRAE-VAR, class-uniform basis
tokenizer rFID1.0940.9550.963
epoch 8 / 167.58 / 5.376.05 / 4.706.28 / 5.36
epoch 40 (before cooldown)4.183.713.39
40+340+3 (40 ep + 3-ep cooldown)3.882.983.00
epoch 200—2.582.55

The two bases reach virtually identical endpoints (3.003.00 vs. 2.982.98 at 40+340+3, and 2.552.55 vs. 2.582.58 at 200 epochs, both surpassing released VAR-d16's 3.153.15), showing that a basis estimated from a small image set serves the staircase tokenizer as well as a class-uniform one; the class-uniform basis converges more slowly through epoch 16 and catches up by epoch 40. In the revision, Sec. 3 states how the basis is estimated, and the main d16 row uses the class-uniform basis so that the main model and the ablations share one basis.

On seeds: following standard practice in ImageNet 256×256256\times 256 generative modeling (e.g., VAR, LlamaGen, MAR, DiT, VFMTok, DINO-Tok), our results are single training runs; a single 200-epoch d16 run costs about 1,200 GPU-hours, making multi-seed runs not feasible given our limited compute budget. Across these two independently trained runs (separate tokenizer, basis, and generator), the post-decay endpoints agree within 0.02 at 40+340+3 and 0.03 at 200 epochs, and two 50k sample sets of one checkpoint differ by at most 0.06.

On the random-basis variant, we agree that it changes both the retained subspace (66.7% vs 88.0% of the variance at r=256r = 256) and the order, so it compares a variance-ordered PCA basis against an unstructured random basis rather than isolating ordering alone.

W2 / Q2 (persistence; other low-pass filters; distribution shift). We extended every Table-4 in the paper ablation variant through the 3-epoch linear learning-rate cooldown (new). VAR-d16, 50k-sample gFID at (cfg =1.2= 1.2, top-p=0.92p = 0.92); the control variants use tokenizers trained for 105k optimizer steps (Table 4's controlled budget in the paper, where the 105k-step PCA reference reads 6.36 / 5.29 at epochs 8 / 16):

ep8ep16ep4040+340+3
RAE-VAR (PCA basis, Fourier resampling)6.285.363.393.00
pooled resampling (area pooling + bicubic)7.996.124.793.84
random orthonormal basis10.376.185.814.02
RAE-spatial-VAR, for reference7.585.374.183.88
  • The gains persist through the cooldown: at 40+340+3 the pooled-resampling variant is 28% and the random-basis variant 34% behind, and each falls back to the level of RAE-spatial-VAR or below it. Both components are needed.
  • Other low-pass filters: our pooled-resampling variant already uses a box low-pass filter (area pooling). Other anti-aliased kernels are untested, and we will note them as alternative filter choices. What distinguishes the Fourier resampling pair (2D FFT low-pass downsampling and zero-padded upsampling) is that down- and upsampling compose to the identity on band-limited maps, leaving no aliased content in the residual.
  • Distribution shift: despite coming from only 8 of 1,000 classes (77% vs. 88% retained variance and mean absolute cosine similarity ∣cos⁡∣=0.08|\cos| = 0.08 on top-32 principal axes), the submitted basis matches the class-uniform basis at both endpoints (W1), demonstrating strong robustness to basis distribution shift. Extending this evaluation to additional datasets is left for future work.

W3 (qualitative evidence; scope). In the revised appendix, sample grids are shown at full resolution (two per row) and now directly compare VQGAN-VAR, RAE-spatial-VAR, and 43-epoch RAE-VAR alongside 200-epoch VAR-d16 on identical classes and seeds. Following standard practice for controlled comparisons against VAR and related next-scale baselines, our experiments focus on class-conditional ImageNet 256×256256\times 256, with broader data domains left for future work.

AuthorsOpenReview ↗

Response to Reviewer p6tZ (2/2)

Q3 (VFMTok, DINO-Tok). Sec. 2.2 cites both only in passing. We will add a discussion of their architectural differences to Sec. 2.2—specifically, both use single-scale raster tokenizers (256 sequential steps with LlamaGen) on frozen DINOv2-L or DINOv3-B encoders (with DINO-Tok applying a top-32 PCA subspace for vector-quantization stability), whereas RAE-VAR constructs a 9-step nested residual hierarchy over a top-256 PCA subspace with band-limited Fourier views for an unchanged VAR generator—and include their final-checkpoint reported numbers in the main generation table (Table 2 in the paper):

ModelEncoderGeneratorStepsFinal gFID
VFMTok-LDINOv2-LLlamaGen-L (343M, 300 ep)2562.79
DINO-Tok-VQ-LDINOv3-BLlamaGen-L (343M)2562.66
RAE-VAR-d16 (ours)DINOv2-SVAR-d16 (310M, 200 ep)92.55

At the closest generator scale (~310M–343M), RAE-VAR achieves the best final gFID (2.55 vs. 2.79 and 2.66; and 3.00 at 40+340+3 epochs, i.e., 40 epochs plus a 3-epoch cooldown) using only 9 autoregressive next-scale forward passes instead of 256 and a 22M-parameter encoder instead of DINOv2-L or DINOv3-B. It similarly achieves the lowest gFID at the ~1B scale (2.16 for RAE-VAR-d24 vs. 2.19 for VFMTok-XXL with classifier-free guidance and 2.44 for DINO-Tok-VQ-XXL). Because encoder, sampling resolution, guidance, and evaluation protocols differ across papers—and a matched-backbone run is not possible without altering the methods (VAR requires a multi-scale residual code rather than a single-scale raster sequence)—VQGAN-VAR (4.88) and RAE-spatial-VAR (3.88) serve as our controlled counterparts.

Reviewer_p6tZOpenReview ↗

Response of the author's rebuttal

I appreciate the author's feedback, and I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion and Final Remarks for AC and Reviewers (1/2)

We thank all reviewers (p6tZ, we2c, r2Zp) for the constructive discussion and the AC for their consideration. Below we summarize recognized strengths, resolved concerns, and our view on novelty, FID scaling, and FDr$^6$.

1. Strengths Recognized Across Reviews

  • Diagnostic insight ("easy to decode ≠\neq easy to predict"): All reviewers valued our finding that low-rank subspaces preserve more decodable content than spatial downsampling, yet rank-only schedules fail in rollout and oracle first-rank tokens restore it (p6tZ, we2c, r2Zp).
  • Practicality & controlled comparison: With unchanged VAR-d16, matched tokenizer size (109M vs. 111M), and fixed sampling, RAE-VAR beats 200-ep VAR-d16 (3.15 gFID) in 43 ep (3.00; 4.65×4.65\times fewer epochs) and reaches 2.55 at 200 ep (p6tZ, we2c, r2Zp).
  • Ablations & clarity: Thorough ablations, failure analysis, and clear presentation (p6tZ, we2c; Presentation 3 from we2c).

2. Weaknesses Resolved During Rebuttal

All major concerns are addressed in the revised PDF (in blue):

  1. PCA basis sensitivity (p6tZ, we2c, r2Zp): A class-uniform training-split basis matches the 8-class basis (3.00 vs. 2.98 at 40+3 ep; 2.55 vs. 2.58 at 200 ep), confirming the 0.88 gain over RAE-spatial-VAR (3.88) (we2c, r2Zp resolved).
  2. Persistence & full-rank control (p6tZ, we2c, r2Zp): Extending Table-4 ablations to 40+3 ep with the full-rank control (3.19) shows removing PCA (4.02) or Fourier (3.84) falls to/below RAE-spatial-VAR (3.88)—both axes are needed.
  3. Framing, Table 3, & main text (we2c, r2Zp): Aligned Abstract, Sec. 1/2.2/4.4, and Conclusion with the breakdown; noted the 105k vs. 200k tokenizer gap; revised Table 3; moved rank-only/oracle diagnosis and basis details to Sec. 3; updated Fig. 2 and grids. r2Zp raised Soundness to 2.
  4. Related work (p6tZ, we2c, r2Zp): Added VFMTok, DINO-Tok, NVG, SpectralAR, NFIG, FVAR (Sec. 2, Table 2). At ~310–343M scale, RAE-VAR-d16 gets 2.55 gFID (9 steps, DINOv2-S) vs. VFMTok-L (2.79) and DINO-Tok-VQ-L (2.66) (256 steps, DINOv2-L/v3-B).

3. On Novelty, FID Scaling, and Multi-Encoder FDr$^6$

The remaining concern (we2c, r2Zp) is that progressive rank yields a smaller raw gFID drop (3.19 →\to 3.00, -0.19) than PCA+Fourier (3.88 →\to 3.19, -0.69).

1. Coupled two-axis novelty: Against 43-ep VQGAN-VAR (4.88), the full gain is 1.88 (4.88 →\to 3.88 RAE; 3.88 →\to 3.19 PCA+Fourier; 3.19 →\to 3.00 progressive rank). Neither PCA alone (pooled: 3.84 gFID / 9.195 FDr$^6$) nor Fourier alone (random: 4.02 / 9.373) beats RAE-spatial-VAR (3.88 / 8.801); gains require coupling both axes in representation space.

2. Raw gFID understates low-FID gains (confirmed by ⋅\sqrt{\cdot} scaling and FDr$^6$): FID is a squared 2-Wasserstein distance:

FID=W22=∥μr−μg∥22+Tr(Σr+Σg−2(ΣrΣg)1/2)\mathrm{FID} = W_2^2 = \lVert \mu_r - \mu_g \rVert_2^2 + \mathrm{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})

Since ΔFID≈2W2ΔW2\Delta\mathrm{FID} \approx 2W_2\Delta W_2, equal distributional gains produce smaller raw drops at lower FID, so linear comparisons should use gFID=W2\sqrt{\mathrm{gFID}} = W_2. Likewise, Appendix B.6 reports FDr$^6$ (Yang et al., 2026), the mean of FD(G,train)/FD(val,train)\mathrm{FD}(G,\mathrm{train})/\mathrm{FD}(\mathrm{val},\mathrm{train}) over 6 frozen encoders (Inception, ConvNeXt-V2, MAE, DINOv2, SigLIP2, CLIP; real val = 1.0). Since each FD\mathrm{FD} is also squared, we compare FDr6\sqrt{\mathrm{FDr}^6} on the same 40+3 ep ablations (staircase Δ\Delta vs. full-rank):

Stage (VAR-d16, 40+3 ep)gFID (Δ\Delta)gFID\sqrt{\mathrm{gFID}} (Δ\Delta)FDr$^6$ (Δ\Delta)FDr6\sqrt{\mathrm{FDr}^6} (Δ\Delta)
VQGAN-VAR4.882.2099.6973.114
RAE-spatial-VAR3.88 (-1.00)1.970 (-0.239)8.801 (-0.896)2.967 (-0.147)
Full-rank (PCA+Fourier, 105k)3.19 (-0.69)1.786 (-0.184)8.116 (-0.685)2.849 (-0.118)
Staircase (105k ref., matched)3.09 (-0.10)1.758 (-0.080)7.642 (-0.474)2.764 (-0.084)
RAE-VAR (staircase, 200k)3.00 (-0.19)1.732 (-0.054)7.680 (-0.436)2.771 (-0.078)

On FDr6\sqrt{\mathrm{FDr}^6}, progressive rank yields 41.8% of the gain over RAE-spatial-VAR (-0.084 of -0.202; 40.9% raw) at a matched 105k budget (2.771 vs. 2.849, matching 200k RAE-VAR at 2.771)—confirming it is neither a budget artifact nor marginal across encoders. On gFID, it also moves the 43-ep model past 200-ep VAR-d16 (3.19 > 3.15 →\to 3.00 < 3.15). We will add these FDr$^6$ results and ⋅\sqrt{\cdot} scaling in the camera-ready.

We thank the reviewers for strengthening our paper and the AC for overseeing the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.