- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the mismatch between the tokenizer and the AR generator in visual autoregressive generation. In the standard setting, a VQ tokenizer is trained for reconstruction alone and then frozen, so it is unaware of which joint token distribution the AR generator finds easier to predict. However, end-to-end training is hindered by the non-differentiable argmax in the quantizer, and the traditional straight-through estimator collapses. The key idea is to let the AR guide the tokenizer through representation alignment rather than the NTP loss, which would collapse the codebook into a few low-entropy codes. Concretely, each position’s codebook assignment is read out twice onto the AR embedding table: a hard one-hot branch trains the AR with NTP and REPA, while a differentiable soft branch, softmax(A/τ)·E, carries a REPA loss against DINOv2 features that updates only the tokenizer. On ImageNet 256×256 and GPIC text-to-image, a fresh AR trained on the frozen GEAR tokenizer consistently outperforms LlamaGen-REPA. With CFG, gFID improves from 6.00 to 4.95 (B), 3.15 to 2.95 (L) and 2.68 to 2.52 (XL), and GPIC FDD improves from 127.9 to 115.3. The fresh AR also reaches the baseline’s final NTP and REPA losses 2.5× and 11.1× faster.
Critical strengths
- The paper gives a principled answer to why naive end-to-end VQ-AR training fails. It guides the tokenizer through an external representation target instead of the NTP loss, which rewards predictability over fidelity, and the STE collapse in Table 5 makes this concrete. This brings end-to-end training to the harder discrete setting.
- The dual hard/soft read-out is simple and general. It leaves the AR and inference unchanged and improves results across three quantizers and on text-to-image generation.
- The most insightful finding is that guidance moves semantic alignment from the tokenizer into the AR, the opposite of diffusion-side recipes. This suggests that the tokenizer’s role can shift from carrying semantics to producing tokens the generator can easily build semantics from.
- The improved tokenizer transfers to a freshly trained AR, which converges much faster. Matched baselines and broad ablations make these gains credible.
Critical weaknesses
-
Missing control for extra tokenizer training. GEAR's tokenizer receives additional training steps, including L_VQ, beyond the frozen warm-up tokenizer used by the baselines. Without a control that continues tokenizer training with the guidance disabled (λ_s = 0) for the same number of steps, it is unclear how much of the improvement comes from the AR guidance itself.
-
"End-to-end" framing vs. the evaluated protocol. All main results train a fresh AR on the frozen GEAR tokenizer, so the evaluated system is effectively two-stage, closer to AR-guided tokenizer fine-tuning. The paper should state this clearly, report the jointly trained AR, and include the compute of the joint stage, since the convergence speed-ups do not account for it.
-
No comparison with Gumbel-softmax or stochastic quantization. These are standard ways to make discrete assignments differentiable and are the natural alternatives to the deterministic soft read-out. Without them, it is hard to tell whether the specific dual read-out design matters or whether any differentiable relaxation would work.
-
Limited evidence of generality and modest gains with CFG. All experiments use raster-order causal AR models, so it is unclear whether the guidance transfers to other AR paradigms such as next-scale or masked generation. Higher resolutions appear only in an ablation. Moreover, on ImageNet with CFG, the gains at L and XL scale are small, and without variance estimates their significance is unclear.
Questions for authors
See weakness.
4 comments
Q5. Does the method remain effective with stochastic relaxations such as Gumbel-softmax?
Response: We added an experiment replacing softmax in the soft branch with Gumbel-softmax, keeping the hard branch and all other training settings unchanged.
| Soft branch | gFID↓ | sFID↓ | IS↑ | Precision↑ | Recall↑ | PSNR↑ | SSIM↑ | rFID↓ |
|---|---|---|---|---|---|---|---|---|
| Softmax (GEAR) | 10.630 | 5.111 | 92.700 | 0.701 | 0.630 | 20.779 | 0.558 | 1.640 |
| Gumbel-softmax | 10.305 | 5.059 | 92.939 | 0.721 | 0.634 | 20.598 | 0.543 | 1.557 |
Gumbel-softmax achieves generation performance comparable to softmax, showing that GEAR also works with stochastic soft relaxations. Under the same Large model, 400k steps, and batch size 256, an additional control passing only REPA gradients to the tokenizer through STE yields gFID 139.124, whereas the two soft read-outs achieve 10.630 and 10.305, respectively.
Q6. Do the tokenizer's benefits transfer beyond raster-order causal AR?
Response: We added experiments with random-order AR (RAR-B) and non-causal masked generation (MaskGIT), testing both direct transfer and target-task guidance.
| Downstream generator | Tokenizer setting | FID↓ | IS↑ | Precision↑ | Recall↑ |
|---|---|---|---|---|---|
| RAR-B | Official baseline | 1.95 | 290.5 | 0.82 | 0.58 |
| RAR-B | LlamaGen-guided GEAR, direct transfer | 1.79 | 292.1 | 0.82 | 0.59 |
| RAR-B | Target-task-guided GEAR | 1.77 | 294.7 | 0.83 | 0.59 |
| MaskGIT (227M) | Official baseline | 6.18 | 182.1 | 0.80 | 0.51 |
| MaskGIT (227M) | LlamaGen-guided GEAR, direct transfer | 4.29 | 188.1 | 0.81 | 0.52 |
| MaskGIT (227M) | Target-task-guided GEAR | 4.23 | 189.4 | 0.81 | 0.53 |
Baseline values are from the official reports; our GEAR experiments follow the respective official generator training and evaluation settings. RAR originally uses MaskGIT's VQ-VAE. Our target-task-guided experiments likewise start from this tokenizer, extending the evaluation to another tokenizer initialization.
Direct transfer of the LlamaGen-guided GEAR tokenizer improves FID by 8.2% for RAR and 30.6% for MaskGIT, demonstrating transfer across generation orders and to non-causal masked generation.
Q7. Can you add full high-resolution experiments beyond ablations?
Response: We added a full generation experiment at resolution 384, using the same training and evaluation settings as LlamaGen and changing only the tokenizer:
| Method | FID↓ | IS↑ | Precision↑ | Recall↑ |
|---|---|---|---|---|
| LlamaGen | 3.42 | 202.93 | 0.79 | 0.56 |
| GEAR tokenizer | 2.18 | 269.12 | 0.85 | 0.61 |
FID improves by 36.3%, and all three other metrics also improve.
The rebuttal period does not allow us to complete a full training experiment at resolution 512; we commit to adding it in a subsequent revision. Table 11 already demonstrates gains at resolution 512 using Large, 100k steps, and batch size 256: gFID without / with CFG improves from 27.742 / 10.204 to 21.826 / 7.595, a reduction of 21.3% / 25.6%. Following the protocol in our submission, this ablation generates images at resolution 512 and resizes them to 256 for FID evaluation.
This ablation budget also matches REPA-E's default ablations and resolution-512 experiment which use Large models, 100k steps, and a global batch size of 256. Together with the new full experiment at resolution 384 and the consistent improvements at resolutions 256, 384, and 512 in our submission, these results support GEAR's generalization across resolutions.
Q8. Could the modest CFG gains at L/XL scale be due to randomness?
Response: We added experiments measuring evaluation and training variability. We evaluate each fixed XL EMA checkpoint with the same three sampling seeds, generating 50k images per seed. Results with CFG are reported as mean ± standard deviation:
| Method | XL gFID↓ |
|---|---|
| LlamaGen-REPA | 2.680 ± 0.007 |
| GEAR | 2.520 ± 0.009 |
The mean difference of 0.160 is substantially larger than the observed sampling variability. Across three independent Base training runs, GEAR achieves gFID 21.514 ± 0.041, consistent with the reported 21.516 and clearly better than the baseline's 24.986 under this setting. All results in the paper use EMA weights with decay 0.9999.
The L/XL gains also persist with longer training: at 300 epochs, gFID with CFG improves from 3.15 / 2.68 to 2.95 / 2.52; at 800 epochs, it improves from 2.92 / 2.57 to 2.72 / 2.45.
Thank you for the detailed response and the additional experiments. The matched-budget tokenizer control, the Gumbel-softmax comparison, and the transfer to RAR and MaskGIT address my main concerns. I encourage the authors to clarify the two-stage protocol in the main text and revise the reconstruction claims, since the unguided control achieves better rFID. I will maintain my positive score.
Thank you for your positive assessment and constructive suggestions. The two-stage evaluation protocol is described in Section 3.1 and Appendix C, and we will make it more prominent in the main text. We will also revise the claims of reconstruction.
Response to Reviewer ea8w
Thank you for recognizing representation-guided optimization of discrete tokenizers, the dual read-out design, and the tokenizer's transferability. We added a matched-budget tokenizer control, Gumbel-softmax experiments, transfer to other generators, multiple-seed experiments, and actual training time measurements. We also clarify the evaluation protocol below.
Q1. Could the gains come from additional tokenizer training rather than AR guidance?
Response: We added a control that continues training the warm-up tokenizer for 400k steps with AR guidance disabled. For the tokenizer, this is equivalent to
λ_s = 0, retaining the original VQ objective. Both groups use the same number of additional tokenizer training steps and the same generator training and evaluation protocols.The unguided control has better reconstruction metrics, yet GEAR reduces generation gFID by 28.9%. Under the same additional training budget, AR representation guidance provides an independent generation benefit.
Q2. Why call the method “end-to-end” when the main results train a fresh AR model on a frozen tokenizer?
Response: “End-to-end” refers to the joint optimization of the tokenizer and AR model. For the main tables, we freeze the tokenizer after joint training and train a fresh generator using the same AR architecture, initialization procedure, and training schedule as the baseline. This controlled comparison highlights the improvements in convergence speed and generation quality brought by the jointly trained tokenizer.
The jointly trained AR model itself also achieves better generation quality: under the same training budget, B/L/XL all outperform LlamaGen-REPA, as reported in Q3. We will clarify the joint training and subsequent evaluation procedures in the main text and table captions.
Q3. Can you report the jointly trained AR model, rather than only the retrained AR model?
Response: Table 7 of our submission already reports the jointly trained AR results. For convenience, we reproduce the comparison at the same 400k steps and batch size 256, without CFG:
The jointly trained AR model outperforms the baseline at all three sizes. We will explicitly identify these joint training results in the main text.
Q4. Does the speed advantage remain after including joint training costs?
Response: Yes. We added total training time comparisons that include approximately 20 hours of joint training. The following XL results use the same hardware configuration and are all without CFG:
Including the joint training cost, GEAR achieves better generation quality with 40% less total training time; at the same 200-hour total budget, it reduces gFID by 15.5%.