- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper replaces the VQGAN tokenizer of VAR with a DINOv2-based RAE and asks what coarse-to-fine should mean in representation space. Channel rank truncation preserves decodable content much better than spatial downsampling, but putting all that content into the first scale makes it hard to predict from short context. From this observation the paper proposes a staircase tokenizer that grows spatial resolution and channel rank together (nested PCA basis + Fourier resampling). At d16 it surpasses the official VAR-d16 with about one-fifth of the generator epochs, and comes close at d20 and d24. The idea is natural, but the key evidence is scattered over the appendix (App. A.3, C.1, C.4, etc.), and the fact that the headline basis spans only 8 classes is disclosed only in App. A.3. So from the paper alone it is hard to tell whether the gain comes from the proposed allocation principle.
Critical strengths
-
The practicality is what strengthens this paper. The backbone is unchanged, the tokenizer size is similar (109M vs. 111M), the released VAR checkpoints are re-evaluated under one pipeline, and the sampler setting is fixed per family. Under these conditions the method outperforms the official VAR-d16 with about one-fifth of the generator epochs. This is the result that makes the paper worth considering.
-
The "easy to decode is not easy to predict" insight. The rank-only schedule reconstructs well but collapses in generation, and the oracle first rank band restores it, as shown in App. C.4. This diagnosis is convincing and informative for next-scale tokenizer design in general, yet it would have been better if it were part of the main text.
Critical weaknesses
-
One critical issue would be the fact that the headline gain cannot be attributed to the proposed principle. Rank allocation itself is never isolated. Tab. 4 ablates only the basis and the resampling inside the staircase, and there is no RAE-spatial-VAR + Fourier resampling result. Since the pooled staircase alone is worse than spatial, the possibility that the whole gain comes from Fourier resampling cannot be excluded. More importantly, the headline model and the ablations use different bases. The headline model uses a 400-image PCA basis that spans only 8 classes, while the ablations use a class-uniform training-split PCA basis. With the class-uniform basis, the staircase versus spatial gap is substantial at ep8 (6.28 vs. 7.58) but nearly vanishes at ep16 (5.36 vs. 5.37). So the headline gap appears only with the 8-class basis. The authors state that they have no explanation for this (App. A.3, C.1), and the front-loading account in Sec. 4.5 points the other way. So it is unknown how much of the 2.98/2.58 in Tab. 2 is attributable to the design. The removal of the step and the choice of the rank schedule are also not justified.
-
The other issue is that every result is from a single run, and the depth claim in Tab. 3 remains unreliable. App. B.5 reports a mid-training checkpoint fluctuation of 3.34 at epoch 55, 3.70 at 70, and 3.41 at 85 for d16, and the paper itself says this is far above the 50k-FID noise floor. Tab. 3 then claims that d20 and d24 approach the released checkpoints with gaps of and , which are well inside that fluctuation. The App. B.5 fluctuation occurs during learning-rate decay, so it is not the same as seed variance, but that is exactly the point. The paper has no per-seed statistics at all, so there is no way to tell whether reflects proximity or noise. The ablations in Tab. 4 rest on single ep8/ep16 checkpoints without cooldown as well.
Questions for authors
-
What happens when only Fourier resampling (and PCA rotation at full rank) is applied to RAE-spatial-VAR? Can you show the independent contribution of rank allocation?
-
What are the 40+3 and 200-epoch results for the class-uniform training-split PCA staircase?
-
How were the rank schedule and the removal of the step decided? Were other schedules tried, for example one that includes the step?
-
Can you report the standard deviation over at least two more seeds for the Tab. 3 depth comparison and the ep16 ablation in Tab. 4, since they show only marginal differences?
8 comments
Response to Reviewer r2Zp (2/2)
W2 / Q4 (single runs; Table 3 in the paper). Following standard practice in ImageNet generative modeling (e.g., VAR, LlamaGen, MAR, DiT, VFMTok, DINO-Tok), our results are single training runs; a single 200-epoch d16 run costs about 1,200 GPU-hours, making multi-seed runs across Tables 3 and 4 in the paper not feasible given our limited compute budget. We did two things instead: every comparison our claims use is now read at a post-decay endpoint (all Table-4 in the paper variants extended to , i.e., 40 training epochs plus a 3-epoch linear cooldown; W1), and we measured the spread at those points.
We agree with the reading of App. B.5 and measured it further. At the same sampling setting (cfg , top-), the class-uniform d16 run shows the same behaviour: 3.39, 3.61, 3.56 and 3.29 at epochs 40, 50, 70 and 100, next to 3.71, 3.34, 3.70, 3.41 and 3.14 at epochs 40, 55, 70, 85 and 100 for the submitted run. While pre-decay checkpoints (at of peak learning rate) fluctuate by 0.2–0.4, post-decay endpoints across two independently trained d16 runs agree within 0.02 at and 0.03 at 200 epochs (with two 50k sample sets of one checkpoint differing by at most 0.06).
Consequently, (i) we now evaluate all Table-4 in the paper ablations at post-decay endpoints (; gaps of 0.84, 1.02, and 0.19), (ii) we revise our Table-3 in the paper wording to "within 0.2 gFID of the released checkpoints with about fewer generator epochs" (explicitly stating that at d20 is parity and at d24 is a deficit), and (iii) we anchor our main claims on the large d16 margins (0.60 against released VAR-d16, 0.88 against RAE-spatial-VAR, and 1.88 against VQGAN-VAR). The d20 () and d24 () cooldown branches start at epochs 50 and 70, inside the fluctuating window; the d16 pair suggests the decay does not carry such deviations into the endpoint, but we have not tested this at d20 and d24.
Presentation. We moved the basis provenance, class-uniform full-budget results, and rank-only failure analysis (with the oracle first-rank-band diagnosis) directly into Sec. 3 of the main paper.
Thank you for the responses and the additional experiments. The class-uniform basis results show that the headline gain does not depend on the 8-class basis, and the revised wording for Tab. 3 now matches what the numbers support. I also appreciate the honest answer on how the rank schedule was decided. However, although the response states that the rank-only failure analysis was moved into Sec. 3, the revised paper still only refers to App. C.4 for it (it would also have been better if the authors had marked where the changes were made).
The full-rank control shows that rank allocation accounts for only 0.19 of the 0.88 gain, while most of it already comes from the PCA projection and Fourier resampling without any rank restriction. I am not arguing that the 0.19 improvement is only marginal, and I acknowledge that the revised Sec. 4.4 and the conclusion now state that the rank schedule has a smaller effect. However, the title, abstract, and introduction still present constrained allocation as the central idea, so the framing is only partly updated.
The proposed method may consequently improve the performance, and since most of my concerns on the experiments have been addressed, I raise my soundness to 2. However, from what I have been given, the main framing of the paper still does not match where the gain comes from, so I keep my initial rating of 2.
Response to Reviewer r2Zp (1/2)
We thank the reviewer for the two strengths named: the result under an unchanged backbone and a single pipeline, and the "easy to decode is not easy to predict" diagnosis. Following the reviewer, we move the rank-only failure and the oracle first-rank-band test (supplying ground-truth tokens for the first rank band during rollout) from App. C.4 into Sec. 3, and the basis estimate and the class-uniform result into the main text.
W1 / Q1 / Q2 / Q3 (attribution; bases; schedule). During the discussion period we ran the full-rank ablation variant requested in Q1 and extended every Table-4 in the paper variant through the cooldown. VAR-d16, 50k-sample gFID; the control variants use tokenizers trained for 105k optimizer steps; each family is evaluated at its fixed sampling setting: