acceptodds
Under review as a conference paper at ICLR 2027

Guided End-to-End AutoRegression for Image Synthesis

Abstract

Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.

Primary area:
generative models
Keywords:
End-to-End, AR Image Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ea8wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between the tokenizer and the AR generator in visual autoregressive generation. In the standard setting, a VQ tokenizer is trained for reconstruction alone and then frozen, so it is unaware of which joint token distribution the AR generator finds easier to predict. However, end-to-end training is hindered by the non-differentiable argmax in the quantizer, and the traditional straight-through estimator collapses. The key idea is to let the AR guide the tokenizer through representation alignment rather than the NTP loss, which would collapse the codebook into a few low-entropy codes. Concretely, each position’s codebook assignment is read out twice onto the AR embedding table: a hard one-hot branch trains the AR with NTP and REPA, while a differentiable soft branch, softmax(A/τ)·E, carries a REPA loss against DINOv2 features that updates only the tokenizer. On ImageNet 256×256 and GPIC text-to-image, a fresh AR trained on the frozen GEAR tokenizer consistently outperforms LlamaGen-REPA. With CFG, gFID improves from 6.00 to 4.95 (B), 3.15 to 2.95 (L) and 2.68 to 2.52 (XL), and GPIC FDD improves from 127.9 to 115.3. The fresh AR also reaches the baseline’s final NTP and REPA losses 2.5× and 11.1× faster.

Critical strengths
  1. The paper gives a principled answer to why naive end-to-end VQ-AR training fails. It guides the tokenizer through an external representation target instead of the NTP loss, which rewards predictability over fidelity, and the STE collapse in Table 5 makes this concrete. This brings end-to-end training to the harder discrete setting.
  2. The dual hard/soft read-out is simple and general. It leaves the AR and inference unchanged and improves results across three quantizers and on text-to-image generation.
  3. The most insightful finding is that guidance moves semantic alignment from the tokenizer into the AR, the opposite of diffusion-side recipes. This suggests that the tokenizer’s role can shift from carrying semantics to producing tokens the generator can easily build semantics from.
  4. The improved tokenizer transfers to a freshly trained AR, which converges much faster. Matched baselines and broad ablations make these gains credible.
Critical weaknesses
  1. Missing control for extra tokenizer training. GEAR's tokenizer receives additional training steps, including L_VQ, beyond the frozen warm-up tokenizer used by the baselines. Without a control that continues tokenizer training with the guidance disabled (λ_s = 0) for the same number of steps, it is unclear how much of the improvement comes from the AR guidance itself.

  2. "End-to-end" framing vs. the evaluated protocol. All main results train a fresh AR on the frozen GEAR tokenizer, so the evaluated system is effectively two-stage, closer to AR-guided tokenizer fine-tuning. The paper should state this clearly, report the jointly trained AR, and include the compute of the joint stage, since the convergence speed-ups do not account for it.

  3. No comparison with Gumbel-softmax or stochastic quantization. These are standard ways to make discrete assignments differentiable and are the natural alternatives to the deterministic soft read-out. Without them, it is hard to tell whether the specific dual read-out design matters or whether any differentiable relaxation would work.

  4. Limited evidence of generality and modest gains with CFG. All experiments use raster-order causal AR models, so it is unclear whether the guidance transfers to other AR paradigms such as next-scale or masked generation. Higher resolutions appear only in an ablation. Moreover, on ImageNet with CFG, the gains at L and XL scale are small, and without variance estimates their significance is unclear.

Questions for authors

See weakness.

4 comments

Response to Reviewer ea8w

Thank you for recognizing representation-guided optimization of discrete tokenizers, the dual read-out design, and the tokenizer's transferability. We added a matched-budget tokenizer control, Gumbel-softmax experiments, transfer to other generators, multiple-seed experiments, and actual training time measurements. We also clarify the evaluation protocol below.

Q1. Could the gains come from additional tokenizer training rather than AR guidance?

Response: We added a control that continues training the warm-up tokenizer for 400k steps with AR guidance disabled. For the tokenizer, this is equivalent to λ_s = 0, retaining the original VQ objective. Both groups use the same number of additional tokenizer training steps and the same generator training and evaluation protocols.

Tokenizer settingPSNR↑SSIM↑rFID↓gFID↓sFID↓IS↑Precision↑Recall↑
Additional 400k steps, no AR guidance21.0310.5691.56114.9416.01276.8340.6630.648
GEAR20.7790.5581.64010.6305.11192.7000.7010.630

The unguided control has better reconstruction metrics, yet GEAR reduces generation gFID by 28.9%. Under the same additional training budget, AR representation guidance provides an independent generation benefit.

Q2. Why call the method “end-to-end” when the main results train a fresh AR model on a frozen tokenizer?

Response: “End-to-end” refers to the joint optimization of the tokenizer and AR model. For the main tables, we freeze the tokenizer after joint training and train a fresh generator using the same AR architecture, initialization procedure, and training schedule as the baseline. This controlled comparison highlights the improvements in convergence speed and generation quality brought by the jointly trained tokenizer.

The jointly trained AR model itself also achieves better generation quality: under the same training budget, B/L/XL all outperform LlamaGen-REPA, as reported in Q3. We will clarify the joint training and subsequent evaluation procedures in the main text and table captions.

Q3. Can you report the jointly trained AR model, rather than only the retrained AR model?

Response: Table 7 of our submission already reports the jointly trained AR results. For convenience, we reproduce the comparison at the same 400k steps and batch size 256, without CFG:

Generator sizeLlamaGen-REPA gFID↓Jointly trained GEAR AR gFID↓
B24.98621.516
L14.71910.630
XL9.6317.693

The jointly trained AR model outperforms the baseline at all three sizes. We will explicitly identify these joint training results in the main text.

Q4. Does the speed advantage remain after including joint training costs?

Response: Yes. We added total training time comparisons that include approximately 20 hours of joint training. The following XL results use the same hardware configuration and are all without CFG:

MethodTotal training timegFID↓
LlamaGen-REPA-XL200 hours7.46
GEAR-XL120 hours (20 hours of joint training + 100 hours of AR training)6.76
GEAR-XL200 hours (20 + 180)6.30

Including the joint training cost, GEAR achieves better generation quality with 40% less total training time; at the same 200-hour total budget, it reduces gFID by 15.5%.

Q5. Does the method remain effective with stochastic relaxations such as Gumbel-softmax?

Response: We added an experiment replacing softmax in the soft branch with Gumbel-softmax, keeping the hard branch and all other training settings unchanged.

Soft branchgFID↓sFID↓IS↑Precision↑Recall↑PSNR↑SSIM↑rFID↓
Softmax (GEAR)10.6305.11192.7000.7010.63020.7790.5581.640
Gumbel-softmax10.3055.05992.9390.7210.63420.5980.5431.557

Gumbel-softmax achieves generation performance comparable to softmax, showing that GEAR also works with stochastic soft relaxations. Under the same Large model, 400k steps, and batch size 256, an additional control passing only REPA gradients to the tokenizer through STE yields gFID 139.124, whereas the two soft read-outs achieve 10.630 and 10.305, respectively.

Q6. Do the tokenizer's benefits transfer beyond raster-order causal AR?

Response: We added experiments with random-order AR (RAR-B) and non-causal masked generation (MaskGIT), testing both direct transfer and target-task guidance.

Downstream generatorTokenizer settingFID↓IS↑Precision↑Recall↑
RAR-BOfficial baseline1.95290.50.820.58
RAR-BLlamaGen-guided GEAR, direct transfer1.79292.10.820.59
RAR-BTarget-task-guided GEAR1.77294.70.830.59
MaskGIT (227M)Official baseline6.18182.10.800.51
MaskGIT (227M)LlamaGen-guided GEAR, direct transfer4.29188.10.810.52
MaskGIT (227M)Target-task-guided GEAR4.23189.40.810.53

Baseline values are from the official reports; our GEAR experiments follow the respective official generator training and evaluation settings. RAR originally uses MaskGIT's VQ-VAE. Our target-task-guided experiments likewise start from this tokenizer, extending the evaluation to another tokenizer initialization.

Direct transfer of the LlamaGen-guided GEAR tokenizer improves FID by 8.2% for RAR and 30.6% for MaskGIT, demonstrating transfer across generation orders and to non-causal masked generation.

Q7. Can you add full high-resolution experiments beyond ablations?

Response: We added a full generation experiment at resolution 384, using the same training and evaluation settings as LlamaGen and changing only the tokenizer:

MethodFID↓IS↑Precision↑Recall↑
LlamaGen3.42202.930.790.56
GEAR tokenizer2.18269.120.850.61

FID improves by 36.3%, and all three other metrics also improve.

The rebuttal period does not allow us to complete a full training experiment at resolution 512; we commit to adding it in a subsequent revision. Table 11 already demonstrates gains at resolution 512 using Large, 100k steps, and batch size 256: gFID without / with CFG improves from 27.742 / 10.204 to 21.826 / 7.595, a reduction of 21.3% / 25.6%. Following the protocol in our submission, this ablation generates images at resolution 512 and resizes them to 256 for FID evaluation.

This ablation budget also matches REPA-E's default ablations and resolution-512 experiment which use Large models, 100k steps, and a global batch size of 256. Together with the new full experiment at resolution 384 and the consistent improvements at resolutions 256, 384, and 512 in our submission, these results support GEAR's generalization across resolutions.

Q8. Could the modest CFG gains at L/XL scale be due to randomness?

Response: We added experiments measuring evaluation and training variability. We evaluate each fixed XL EMA checkpoint with the same three sampling seeds, generating 50k images per seed. Results with CFG are reported as mean ± standard deviation:

MethodXL gFID↓
LlamaGen-REPA2.680 ± 0.007
GEAR2.520 ± 0.009

The mean difference of 0.160 is substantially larger than the observed sampling variability. Across three independent Base training runs, GEAR achieves gFID 21.514 ± 0.041, consistent with the reported 21.516 and clearly better than the baseline's 24.986 under this setting. All results in the paper use EMA weights with decay 0.9999.

The L/XL gains also persist with longer training: at 300 epochs, gFID with CFG improves from 3.15 / 2.68 to 2.95 / 2.52; at 800 epochs, it improves from 2.92 / 2.57 to 2.72 / 2.45.

Thank you for the detailed response and the additional experiments. The matched-budget tokenizer control, the Gumbel-softmax comparison, and the transfer to RAR and MaskGIT address my main concerns. I encourage the authors to clarify the two-stage protocol in the main text and revise the reconstruction claims, since the unguided control achieves better rFID. I will maintain my positive score.

Thank you for your positive assessment and constructive suggestions. The two-stage evaluation protocol is described in Section 3.1 and Appendix C, and we will make it more prominent in the main text. We will also revise the claims of reconstruction.

Reviewer_RorCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, authors propose the framework for jointly training a discrete tokenizer and autoregressive models with representation guidance. In particular, this paper suggests dual branch forward which the hard branch teaches the generator using next-token prediction and representation alignment and soft branch makes alignment gradients to the tokenizer. As a result, paper shows faster convergence in training and improved generation quality.

Critical strengths
  1. Well-designed formulation of representation-guided joint training
  • The dual branch shows a clear principle for adapting a tokenizer using representation guidance from the generator. In particular, the hard branch trains the generator on the discrete tokens, while the soft branch propagates alignment gradients to update the tokenizer. The update rules depicted in Section 2.1 and Algorithm 1 identify the roles of the two branches.
  1. Empirical results across diverse experimental settings.
  • Comparison with LlamaGen-REPA demonstrate improvements across model sizes. Also, the ablation studies show a variety of quantization schemes and encoders. In Table 6, authors provide enhancement in both gFID and rFID for VQ-VAE, LFQ and IBQ. Those experimental results demonstrate the effectiveness of the proposed approach.
Critical weaknesses
  1. Generality beyond the LlamaGen models remains insufficiently demonstrated
  • In Table 1, proposed method provides the performance of the two different methods within one family: LlamaGen and LlamaGen-REPA. Therefore, it is still unclear whether the proposed framework still can help other VQ-based autoregressive models such as RAR [1]. Applying proposed approach to an additional model and comparing its effectiveness would enhance the empirical aspect for the generality of the approach.
  1. Insufficient ablation for soft branch technique
  • In Table 5, the ablation studies are constrained within only STE-based training and removal of the adversarial loss. Although Table 4 varies the alignment coefficient shared by both the hard and soft branches, the ablation experiments do not directly isolate the effect of the soft representation alignment to training tokenizer. More specifically, it still remain unclear how much of improvement is induced by the proposed soft branch. Fair comparison that removes the soft branch loss while maintaining the hard branch and continuing tokenizer training under the same training budget would help identify the contribution of the proposed method.

[1] Qihang Yu et al, Randomized Autoregressive Visual Generation

Questions for authors
  1. How much training time does proposed method really save?
  • In Figure 1, the reduction in training steps is promising. Could the authors also report the time per iteration and the wall-clock time required for this method and LlamaGen-REPA to reach the same gFID under comparable hardware and batch setting? This might be helpful to clarify the practical speed gain.
2 comments

Response to Reviewer RorC

Thank you for recognizing the dual-branch design and our experiments. To address generality, the soft branch's contribution, and practical speed gains, we added experiments with other generators, a matched-budget control, and total training time comparisons that include the joint stage.

Q1. Do GEAR's benefits generalize beyond LlamaGen, for example to RAR?

Response: We added experiments with RAR-B and MaskGIT, covering random-order autoregressive generation and non-causal masked generation, respectively. We test two settings: directly transferring a GEAR tokenizer checkpoint trained under LlamaGen guidance, and applying GEAR guidance under the target generation task. In both cases, we then freeze the tokenizer and train the downstream generator.

Downstream generatorTokenizer settingFID↓IS↑Precision↑Recall↑
RAR-BOfficial baseline1.95290.50.820.58
RAR-BLlamaGen-guided GEAR, direct transfer1.79292.10.820.59
RAR-BTarget-task-guided GEAR1.77294.70.830.59
MaskGIT (227M)Official baseline6.18182.10.800.51
MaskGIT (227M)LlamaGen-guided GEAR, direct transfer4.29188.10.810.52
MaskGIT (227M)Target-task-guided GEAR4.23189.40.810.53

Baseline values are from the official reports; our GEAR experiments follow the respective official generator training and evaluation settings. RAR originally uses MaskGIT's VQ-VAE. Our target-task-guided experiments likewise train GEAR starting from this tokenizer, extending the evaluation to a tokenizer initialization different from that in our main experiments.

Direct transfer improves FID by 8.2% for RAR-B and 30.6% for MaskGIT, demonstrating that the GEAR tokenizer transfers across generation orders and to masked generation. The target-task-guided variants achieve 1.77 and 4.23, respectively.

Q2. Can continuing tokenizer training for the same number of steps, without soft-branch guidance, yield the same gains?

Response: We added a tokenizer control trained for an additional 400k steps without AR guidance. For the tokenizer, this is equivalent to setting the soft-branch coefficient to λ_s = 0: the original reconstruction, perceptual, adversarial, and quantization objectives are retained, while generator alignment supervision is removed. Both groups use the same generator training and evaluation protocols to isolate AR guidance from additional tokenizer training.

Tokenizer settingPSNR↑SSIM↑rFID↓gFID↓sFID↓IS↑Precision↑Recall↑
Warm-up tokenizer + 400k steps, no AR guidance21.0310.5691.56114.9416.01276.8340.6630.648
GEAR, same additional training steps20.7790.5581.64010.6305.11192.7000.7010.630

The unguided control has better reconstruction metrics, yet GEAR reduces generation gFID by 28.9%. Under the same additional training budget, soft-branch representation guidance provides an independent generation benefit.

Q3. How much actual training time does GEAR save?

Response: We added a comparison of total training time for XL models under the same hardware configuration. All results below are without CFG. GEAR's time includes approximately 20 hours of joint training, followed by training a fresh AR model on the frozen tokenizer:

MethodTotal training timegFID↓
LlamaGen-REPA-XL200 hours7.46
GEAR-XL120 hours (20 + 100)6.76
GEAR-XL200 hours (20 + 180)6.30

Including the joint training cost, GEAR achieves better generation quality with 40% less total training time; at the same 200-hour total budget, it reduces gFID by 15.5%.

Thank you again for your thoughtful feedback. We would appreciate hearing whether our responses and additional experiments have addressed your main concerns. If any points remain unclear or need further attention, we would be happy to clarify them and improve the manuscript accordingly.

Reviewer_j3e7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GEAR trains a VQ tokenizer and an autoregressive generator jointly. To get around the non-differentiable codebook index, GEAR reads the assignment two ways. A hard one-hot lookup trains the AR model with next-token prediction (NTP) and REPA alignment to DINOv2, while a soft read-out runs through the AR up to the alignment layer and sends only the REPA loss back to the tokenizer, so NTP never reaches it. For the main results, the jointly trained tokenizer is frozen and a fresh AR model is trained on it. Experimental results cover class-conditional ImageNet generation at B, L and XL scales, tokenizer reconstruction, ablations across choices of different quantizers and alignment encoders, and a text-to-image study.

Critical strengths
  1. The guided end-to-end tokenizer training strategy is an interesting and less-explored idea.

  2. The gain of the proposed method is consistent across many controlled comparisons. Under controlled experimental settings, GEAR wins at three model sizes, two schedules, three quantizers, four alignment targets, three resolutions and also in the text-to-image setting.

Critical weaknesses

Many competitive baselines are not compared to in the paper. Table 1 compares only with LlamaGen, LlamaGen-REPA and diffusion models. Several recent discrete AR systems seem to report better gFID at similar or smaller generator sizes. The literature I find includes: GigaTok [1], FlexTok [2], Semanticist [3], ResTok [4], and AliTok [5]. I understand that the training recipes differ, so these are not controlled comparisons. Still, the paper should include them in Table 1 and state where GEAR stands.

[1] Xiong et al. GigaTok: Scaling visual tokenizers to 3 billion parameters for autoregressive image generation. arXiv:2504.08736, 2025.

[2] Bachmann et al. FlexTok: Resampling images into 1D token sequences of flexible length. ICML, 2025.

[3] Wen et al. "Principal components" enable a new language of images. arXiv:2503.08685, 2025.

[4] Zhang et al. ResTok: Learning hierarchical residuals in 1D visual tokenizers for autoregressive image generation. arXiv:2601.03955, 2026.

[5] Wu et al. AliTok: Towards sequence modeling alignment between tokenizer and autoregressive model. arXiv:2506.05289, 2025.

Questions for authors
  1. In the STE baseline, which losses reach the tokenizer? What happens when STE carries REPA only?

  2. Can the authors compare GEAR with recent AR generator baselines in Table 1, at matched generator sizes, with and without CFG? Where training recipes differ, a short note on the differences would help readers judge where GEAR stands.

5 comments

Response to Reviewer j3e7

Thank you for recognizing GEAR's training strategy and its consistent gains across controlled settings. We have added comparisons with recent methods and clarified the gradient paths in the STE baseline. We address each concern below.

Q1. Table 1 omits recent competitive AR methods. How does GEAR compare with them?

Response: Thank you for the suggestion. We will add all five cited works to Table 1, specifying differences in generator size, tokenizer, and generation approach. The table below summarizes representative published results, followed by the XL results from our submission for direct comparison:

MethodGenerator sizew/o CFG↓w/ CFG↓Key configuration differences
GigaTok-XL-XXL tokenizer + AR-B111M—3.152.9B tokenizer; 256 tokens; AR trained for 300 epochs
FlexTok d18-d281.33B¹1.71²—32-token setting in Appendix Table 9; flow decoder additionally uses 25 decoding steps and APG
SemanticistεLlamaGen-L³—2.5732 continuous tokens; diffusion prediction head and DiT decoder
ResTok326M—2.34662M tokenizer; 128 tokens; 9-step hierarchical generation; also reports 2.18 with vanilla AR
AliTok-XL662M1.881.28272 tokens; AR trained for 400 epochs; tencrop, batch size 2048, and specialized sampling settings
LlamaGen-XL (Table 1 of our submission)775M15.54⁴3.47⁴300 epochs; inference at resolution 384, as marked in the submission
LlamaGen-REPA-XL775M8.202.68300 epochs; resolution 256; matched baseline for GEAR
GEAR-XL (ours)775M6.762.52300 epochs; resolution 256; same AR training and evaluation settings as LlamaGen-REPA-XL

¹ FlexTok reports non-embedding parameters. ² Only the AR model omits CFG; the flow decoder still uses APG, so the full system is not guidance-free. ³ Semanticist uses continuous representations and additional prediction modules, and is therefore not a parameter-matched configuration of standard discrete LlamaGen-L. ⁴ Following the asterisk in Table 1 of our submission: inference at resolution 384, followed by resizing to 256 for evaluation; the other rows report ImageNet-256 settings. — indicates that the cited table does not report the corresponding value.

GEAR contributes a joint training strategy for the tokenizer and generator. Using the general LlamaGen architecture and the same AR training settings, GEAR-XL improves LlamaGen-REPA-XL's gFID from 8.20 / 2.68 to 6.76 / 2.52 without / with CFG. Differences in tokenizers, generation approaches, and training configurations are listed above.

We also added RAR-B and MaskGIT experiments to assess the strategy across generation frameworks:

Downstream generatorOfficial baseline FID↓Direct transfer of LlamaGen-guided GEAR tokenizer FID↓Target-task-guided GEAR tokenizer FID↓
RAR-B1.951.791.77
MaskGIT (227M)6.184.294.23

Baseline values are from the official reports; our GEAR experiments follow the respective official generator training and evaluation settings. Directly transferring the LlamaGen-guided tokenizer improves both random-order AR and masked generation, and the target-task-guided variants also outperform their baselines. These results further support GEAR's applicability across generation settings.

Q2. Which losses reach the tokenizer in the STE baseline?

Response: The STE baseline in Table 5 (gFID 104.932) corresponds to Figure 2(b): the tokenizer receives its original VQ objective and gradients from NTP + REPA through STE. In GEAR, the tokenizer receives the original VQ objective and REPA guidance through the soft branch; NTP updates only the AR model.

Q3. What happens when STE carries only REPA, without NTP?

Response: We added a control that backpropagates only REPA through STE to the tokenizer, blocking NTP gradients. Generation quality still degrades severely. This control retains the same original VQ objective, Large model, 400k steps, and batch size 256. Only NTP gradients to the tokenizer are blocked; the AR model is still trained with NTP + REPA.

Tokenizer guidanceGradients from ARgFID↓sFID↓IS↑Precision↑Recall↑
Original STE baselineNTP + REPA104.93231.93312.4180.2920.233
New STE controlREPA only, no NTP139.12436.13910.9510.2440.219
GEARREPA only, no NTP10.6305.11192.7000.7010.630

STE with REPA alone still performs poorly; GEAR's soft read-out reduces gFID from 139.124 to 10.630. This supports the effectiveness of the soft read-out design.

Q4. Can you compare methods at matched or similar generator sizes, with and without CFG, and explain recipe differences?

Response: Yes. The following 300-epoch comparisons strictly match the generator, training schedule, and sampling settings. Each pair differs only in whether the frozen tokenizer has undergone GEAR-guided training:

Generator sizeMethodw/o CFG gFID↓w/ CFG gFID↓
B, 111MLlamaGen-REPA20.166.00
B, 111MGEAR16.964.95
L, 343MLlamaGen-REPA12.703.15
L, 343MGEAR8.662.95
XL, 775MLlamaGen-REPA8.202.68
XL, 775MGEAR6.762.52

Among external methods, GigaTok's 111M AR matches GEAR-B in size, but its best result uses a 2.9B tokenizer. AliTok-L's 318M AR is similar in size to GEAR-L's 343M and reports 2.09 / 1.49 without / with CFG at 400 epochs, but differs in tokenizer, token sequence, batch size, augmentation, and sampling strategy. We will specify these differences in the table notes.

GEAR improves generation quality at all three model sizes, both with and without CFG, and the gains persist at 800 epochs. For example, XL gFID at 800 epochs improves from 7.46 / 2.57 to 6.28 / 2.45 without / with CFG.

Thank you again for your thoughtful feedback. We would appreciate hearing whether our responses and additional experiments have addressed your main concerns. If any points remain unclear or need further attention, we would be happy to clarify them and improve the manuscript accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.