- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FeaTok, a discrete image tokenizer for image generation in which the discrete representation is shaped only by reconstruction losses to pretrained visual features rather than by pixel reconstruction. A feature decoder reconstructs multi-layer features from frozen SigLIP, SigLIP2, DINOv2, or DINOv3 encoders, while a pixel decoder is trained on stop-gradient tokens solely to render generated token sequences back into images. The method also discretizes the encoder’s global representation and generates this global token before the patch tokens, providing image-level context for autoregressive generation. Experiments on ImageNet 256×256 and 512×512 use BAR-style autoregressive generators with masked-bit modeling and compare against discrete and continuous baselines using gFDr6, gFID, reward-model scores, and sampling-cost measurements. The authors report that FeaTok improves over prior discrete models, reaches competitive quality with much lower sampling cost than continuous generators, and transfers to text-to-image generation after limited fine-tuning. Ablations study pixel, feature, and mixed supervision; feature depth; global-token augmentation; reconstruction quality; and token-structure statistics.
Critical strengths
Strong controlled empirical study of tokenizer objectives. The paper systematically compares pixel-only, feature-only, and mixed pixel-feature supervision across four pretrained encoders, while largely keeping the BAR-style generator fixed. This makes the central empirical claim—that feature-only supervision can improve generative learnability—more credible than a single-model SOTA comparison.
Strong quality-compute tradeoff. FeaTok-L-DINOv3 reaches gFDr6 3.13 on ImageNet 256×256 using only 1.27 TFLOPs per image, compared with RAEv2 at 3.95 using 47.72 TFLOPs without guidance. This substantial sampling-cost advantage is practically important for discrete image generation and multimodal systems, even though RAEv2 remains stronger under full guidance.
Critical weaknesses
-
Shaping tokens with pretrained visual features has been explored in prior work such as DINO-Tok and TA-Tok, and the global-token idea has also appeared in Holistic Tokenizer. Thus, the contribution of this paper is more like a strong combination of existing ideas. The novelty is incremental and not sufficiently isolated from close prior work.
-
The DINO/SigLIP encoders and reward models were pretrained on large external datasets, which may overlap with the evaluation images or prompts.
-
More methods that use features from pretrained encoders as auxiliary training signals should be added, such as DINO-Tok and TA-Tok, to demonstrate the advantage of the proposed “feature-only objectives” over these methods. Comparisons should include both the main experiments and downstream text-to-image generation.
Questions for authors
The experiments show that although FeaTok achieves overall SOTA performance, it is weaker on the Inception and SigLIP2 sub-metrics, especially in the experiments corresponding to Tables 3 and 4. Have the authors analyzed this issue?
7 comments
Response to reviewer iVs1 (Part 2/5)
W2: Overlap in evaluation
We acknowledge that overlap between external pretraining data and evaluation images or prompts may exist and cannot be easily ruled out, and we will clarify this limitation in the revision. Our use of pretrained visual representations follows common practice in the literature. Our baselines also use pretrained DINO or SigLIP representations, including BAR [6], REPA [7], RAE [8], and RAEv2 [9]. FeaTok achieves lower gFDr6 than BAR-L and RAE at both 256×256 and 512×512, with and without guidance. FeaTok also achieves better reward scores compared to these baselines.
We also tried to minimize the bias introduced by the data overlaps from pretrained visual features.
First, we report three human-preference reward models with different backbones. PickScore and HPSv2 are fine-tuned from CLIP, while HPSv3 uses Qwen2-VL-7B. FeaTok-L-DINOv3 scores higher than BAR-L, RAE, and RAEv2 on all three metrics (Fig. 5 of the manuscript). In particular, its HPSv3 score is 4.38, compared with 3.69, 3.76, and 4.31, respectively, showing that the preference gains also hold with a different reward-model backbone.
Second, we report gFDr4 by excluding DINOv2 and SigLIP2 from the six-space average and expand the comparison in Tables iVs1-1a and iVs1-1b. At 256×256, FeaTok-L-DINOv3 has the lowest unguided gFDr4 among the listed methods. With guidance, it remains better than BAR, REPA, REPA-E, and RAE, while RAEv2 is better similar to the gFDr6 metric. At 512×512, FeaTok-L-DINOv3 has the lowest gFDr4 among the available comparisons in both guidance modes. Thus, the advantage over BAR and RAE persists after removing DINOv2 and SigLIP2 from evaluation. We will revise the manuscript to include a discussion.
Table iVs1-1a. ImageNet-256 gFDr4, averaging the normalized Fréchet distances in Inception-v3, ConvNeXt V2, MAE, and CLIP spaces. Sampling configurations follow manuscript Tables 1 and 2. Guided continuous-baseline scores use the published per-space values in [9]. Lower is better. Bold marks the best score in each column.
| Model | Without guidance ↓ | With guidance ↓ |
|---|---|---|
| BAR-B [6] | 4.07 | 3.85 |
| BAR-L [6] | 3.67 | 3.37 |
| SiT-XL/2-REPA [7] | 9.20 | 4.64 |
| REPA-E-XL/1 | 3.68 | 2.70 |
| RAE-XL [8] | 4.32 | 3.41 |
| RAEv2-XL [9] | 3.91 | 2.32 |
| FeaTok-B-DINOv3 | 2.92 | 2.77 |
| FeaTok-L-DINOv3 | 2.57 | 2.57 |
Table iVs1-1b. ImageNet-512 gFDr4 in the same four feature spaces, using manuscript Table 4 sampling configurations. Lower is better. Bold marks the best score in each column.
| Model | Without guidance ↓ | With guidance ↓ |
|---|---|---|
| BAR-L [6] | 2.76 | 2.39 |
| RAE-XL [8] | 2.92 | 2.18 |
| FeaTok-L-DINOv3 | 1.74 | 1.51 |
Response to reviewer iVs1 (Part 3/5)
W3: Additional baselines
Thank you for pointing us to more baselines. Table iVs1-2 compares the methods at 256×256, without and with guidance. We use Hita’s reported FID and DINO-Tok’s published gFID. We computed Hita and DINO-Tok’s gFDr6 with the authors’ sampler and the released weights. We also invite the reviewer to consult Table fGxQ-2, where we added AliTok, EOSTok, and other gFID baselines in response to another reviewer’s request.
Table iVs1-2. Class-conditional ImageNet-256 comparisons with Hita and DINO-Tok. Paired entries are without / with guidance. Hita and DINO-Tok gFID are reported results, while Hita and DINO-Tok gFDr6 are measured by us with released weights and samplers. Lower is better.
| Model | Generator params | gFID ↓ | gFDr6 ↓ |
|---|---|---|---|
| Hita-V-2B, paper | 2.0B | — / 2.59 | 12.95 / 6.54 |
| Hita-U-2B, repository | 2.0B | — / 1.82 (1.69 with CFG-free guidance) | 4.35 / 3.87 |
| DINO-Tok VQ-XXL | 1.4B reported | — / 2.44 | 16.33 / 10.08 |
| FeaTok-B-DINOv3 | 0.4B | 1.34 / 1.07 | 3.58 / 3.39 |
| FeaTok-L-DINOv3 | 1.1B | 1.15 / 1.05 | 3.13 / 3.08 |
For TA-Tok, there are no results reported using the class-conditional ImageNet setup, and no checkpoint of this ImageNet setup is released. We used their released tokenizer and trained a base-size generative model on 169-token sequences using the same 250K-step recipe as our patch-only ablation. Since they use a VQ-style discretizer instead of FSQ as ours, we tested both a categorical head and a bitwise head. Table iVs1-3 reports the best measured 50K-sample result for each head. The categorical head reaches gFID 5.27 and gFDr6 7.95, while the bitwise head reaches 6.94 and 10.00. FeaTok-DINOv3 with the patch-only BAR-B generator achieves 1.48 and 4.28.
Table iVs1-3. Class-conditional ImageNet-1k generative performance of TA-Tok following our training recipe, compared with patch-only FeaTok-DINOv3. Both TA-Tok rows use de-tokenizer CFG 4.0 following the released setting. The categorical head uses T=1.0, top-k=1200 and top-p=0.95. The bitwise head uses T=0.5 and generator CFG 5.0. Both are tuned during inference. Lower gFID and gFDr6 are better.
| Model | Generator + generative de-tokenizer | Guidance | gFID ↓ | gFDr6 ↓ |
|---|---|---|---|---|
| TA-Tok + BAR-B, categorical head | 0.3B + 0.65B | De-tokenizer only | 5.27 | 7.95 |
| TA-Tok + BAR-B, bitwise head | 0.4B + 0.65B | Generator + de-tokenizer | 6.94 | 10.00 |
| FeaTok-DINOv3 + BAR-B, patch-only | 0.4B + N/A | Off | 1.48 | 4.28 |
We also evaluated reconstruction on all 50K ImageNet validation images using the reconstruction evaluation protocol of Table 12 in our manuscript and TA-Tok’s released 256 px de-tokenizer at CFG 4.0. Table iVs1-4 shows that FeaTok reconstructs more faithfully even when TA-Tok uses its native, denser token grid.
Table iVs1-4. ImageNet-256 reconstruction on 50K validation images using the protocol of Table 12 in our manuscript. TA-Tok uses its released de-tokenizer at CFG 4.0 and either 169 or 729 tokens. Lower rFID and rFDr6, and higher PSNR and SSIM, are better.
| Tokenizer | Tokens | rFID ↓ | rFDr6 ↓ | PSNR ↑ | SSIM ↑ |
|---|---|---|---|---|---|
| TA-Tok | 169 | 4.92 | 7.00 | 13.00 | 0.318 |
| TA-Tok | 729 | 3.72 | 6.36 | 15.56 | 0.417 |
| FeaTok-DINOv3 | 256 | 0.99 | 2.08 | 17.45 | 0.424 |
For text-to-image generation, FeaTok leads BAR and RAEv2 after fine-tuning (Table 5 of our manuscript). As for Tar, the t2i model released with TA-Tok, Table iVs1-5 shows that our method also exceeds Tar-1.5B and approaches Tar-7B on both shared benchmarks, using a smaller tokenizer-training corpus and a direct pixel decoder. We are training a matched TA-Tok model using FeaTok’s generator backbone, data, and schedule, and will post its results if they are ready before the end of the discussion.
Table iVs1-5. Text-to-image system comparison on GenEval and DPG-Bench. Tar scores are reported results. FeaTok scores are after fine-tuning. Tokenizer, de-tokenizer, and generator training data differ across systems. Higher scores are better.
| System | Tokenizer / generative de-tokenizer data | Generator training data | GenEval ↑ | DPG-Bench ↑ |
|---|---|---|---|---|
| Tar-1.5B, reported | 200M / 50M LAION images | 100M mixed samples, including 23M text–image pairs | 0.76 | 82.96 |
| Tar-7B, reported | 200M / 50M LAION images | 100M mixed samples, including 23M text–image pairs | 0.84 | 84.19 |
| FeaTok, BAR-L + Qwen3-0.6B | 1.28M ImageNet images / — | ~36M BLIP-3o images (Long-Caption, Short-Caption, JourneyDB), then BLIP-3o-60K | 0.837 | 83.4 |
Response to reviewer iVs1 (Part 1/5)
We thank the reviewer for recognizing our controlled tokenizer study and FeaTok’s quality–compute trade-off, achieving the SOTA performance.
We address the concerns of the reviewer in the following bullet points.
W1: Novelty
We would like to use the rebuttal to clarify our novelty. FeaTok argues that pixel losses should not shape the discrete tokens during the tokenizer training for image generation. Feature losses but not pixel losses should be the loss to shape the discrete tokens, and a separate pixel decoder is trained with pixel losses using detached discrete tokens. This approach is different from the prevailing design choice shared by recent models, including DINO-Tok [2] and BAR, which assumes image reconstruction losses are beneficial to train useful discrete tokens for generation. TA-Tok [3] uses feature supervision alone to train its semantic tokens, but for the generation task, it still employs a separate generative de-tokenizer to obtain VQ-VAE style discrete tokens shaped by pixel losses. Our work demonstrates that pixel losses are not necessary and can be detrimental to shape discrete tokens in image generation tasks where we care about image details not only semantics. We find the results surprising that achieving state-of-the-art image generation from discrete tokens does not require the tokenizer encoder to leverage pixel-level reconstruction losses.
We do not claim the general idea of global conditioning as novel. As acknowledged in the manuscript, prior work already follows global-to-local generation strategies. We will revise the manuscript to make this distinction explicit. Our main contribution is learning discrete tokens through feature losses without allowing pixel reconstruction gradients to shape them. Global conditioning is a natural addition to this framework because the pretrained visual encoder already provides an image-level representation. RCG [10] and DLC [11] demonstrate the benefits of representation conditioning, while FlexTok [12] uses a related coarse-to-fine ordering of visual tokens. Our implementation requires little additional machinery. Hita [1] learns holistic representations through attention between learnable queries, VQGAN patch embeddings, and DINOv2 features, then fuses the quantized holistic and patch representations before image decoding. FeaTok instead directly quantizes the pretrained encoder’s existing global feature with one additional quantizer. The resulting token conditions patch generation, while the image decoder receives only the patch codes. At the system level, FeaTok-L achieves lower reported gFID with a smaller generator than both Hita variants listed in Table iVs1-2.