- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces GIST (Guided Image Semantic Tokenizer), an adaptive tokenizer that exceeds prior designs that are constraint by pixel reconstruction targets and achieves semantic protection via adding constraints related to semantic reconstruction. Controlled comparison on ImageNet-100 indicates that GIST outperforms pixel-controlled baseline (KARL) with better LPIPS and DreamSim.
Critical strengths
- The observation that pixel-level target is not always suited for adaptive tokenization design is a valuable insight, and that semantic protection provides new insight into designing better adaptive tokenizers.
Critical weaknesses
-
Limited Novelty: The contribution of this paper is straightforward, adding a semantic reconstruction target and semantic conditioning module/signal to control semantic levels. However, adding semantic targets in one or another way has already became the de facto standard practice in tokenizer designs with many prior works (e.g. [1-3]). The authors should further clarify the novelty of this design beyond introducing semantic loss/conditioning based on vision foundation models.
-
Limited evaluation. This limitation is folded. (i) the authors only compared against KARL in the main paper; (ii) the main experiments are on ImageNet-100, the authors should at least perform experiments on larger-scale ImageNet-1k, which is relatively standard for developing visual tokenizers.
-
Unclear generation performance. As the authors stated in related works, the major aim for developing image tokenizers should be to better serve downstream generative models. As discovered by previous work, better reconstruction performance doesn't necessarily translates to better generation performance. Therefore the lack of results on generative models significantly undermine the authors' contribution of the new proposed tokenizer.
[1] Xiong, Tianwei, et al. "Gigatok: Scaling visual tokenizers to 3 billion parameters for autoregressive image generation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
[2] Zheng, Anlin, et al. "Vision foundation models as effective visual tokenizers for autoregressive image generation." arXiv preprint arXiv:2507.08441 (2025).
[3] Fu, Zixuan, et al. "Improving Flexible Image Tokenizers for Autoregressive Image Generation." arXiv preprint arXiv:2601.01535 (2026).
Questions for authors
See weaknesses.
6 comments
Response to Reviewer 2z8g (Part 2/3)
Broader tokenizer comparisons and ImageNet-1k evaluation
(i) Comparing GIST with additional tokenizer families
[…] the authors only compared against KARL in the main paper […]
We agree that the broader comparisons should be more visible in the main text. Our main experiments focus on KARL† because our central question is whether adding semantic control improves adaptive token allocation under a matched tokenizer backbone and training recipe. Comparing GIST with KARL† isolates the effect of our contribution, because they share the base tokenizer, training data, optimization schedule, and evaluation protocol. Comparisons between independently trained tokenizer families cannot isolate this contribution because several of these factors differ simultaneously. For broader context, Appendix C compares GIST with ALIT, three FlexTok variants, three One-D-Piece variants, and the released KARL checkpoint. The table below presents a compact summary of these results at an input budget of 128 tokens.
| Tokenizer | Training data | LPIPS ↓ | DreamSim ↓ |
|---|---|---|---|
| KARL† | IN100 | 0.19 | 0.15 |
| GIST | IN100 | 0.19 | 0.15 |
| Released KARL | IN100 | 0.19 | 0.16 |
| ALIT | IN100 | 0.20 | 0.14 |
| FlexTok, three variants | IN1k | 0.25–0.27 | 0.16–0.17 |
| One-D-Piece, three variants | IN1k | 0.20–0.21 | 0.12–0.14 |
At an input budget of 128 tokens, GIST matches the LPIPS of KARL† and the released KARL checkpoint (0.19), and achieves lower LPIPS than ALIT (0.20), FlexTok (0.25–0.27), and One-D-Piece (0.20–0.21). The DreamSim comparison is mixed: GIST (0.15) performs better than FlexTok (0.16–0.17), while ALIT (0.14) and One-D-Piece (0.12–0.14) achieve lower errors. These results place GIST competitively among existing tokenizers for fixed-budget reconstruction. Appendix C contains a fuller discussion of these results.
We will bring a compact summary of these comparisons into the main text.
(ii) Reconstruction gains extend to ImageNet-1k
[…] the main experiments are on ImageNet-100, the authors should at least perform experiments on larger-scale ImageNet-1k, which is relatively standard for developing visual tokenizers.
Our main reconstruction experiments follow the ImageNet-100 evaluation setting used in KARL’s main reconstruction comparisons (Tables 1–2), our closest predecessor. We have (i) extended this evaluation to the full 50,000-image ImageNet-1k validation set, and (ii) continued stage-2 training of GIST and KARL† on ImageNet-1k under matched optimization settings. Both experiments are reported in Appendix “Reconstruction Quality on ImageNet-1k” (Figure 30 and Table 8).
First, we evaluate the ImageNet-100-trained checkpoints without additional training. GIST retains its reconstruction advantage on ImageNet-1k: at 128 mean active tokens, it achieves LPIPS of 0.232 versus 0.297 for KARL†, closely matching the corresponding ImageNet-100 results of 0.231 versus 0.296. Although DreamSim increases for both methods on ImageNet-1k, the absolute gap remains similar: 0.221 versus 0.298, compared with 0.200 versus 0.278 on ImageNet-100.
Second, we continue stage-2 training of both checkpoints for 20 epochs on ImageNet-1k using matched optimization settings. GIST’s advantage increases at 128 mean active tokens: it achieves LPIPS of 0.215 and DreamSim of 0.194, compared with 0.294 and 0.292 for KARL†, corresponding to reductions of 26.9% and 33.5%, respectively. The advantage also persists at 192 mean active tokens. These comparisons use linear interpolation at matched mean active token counts.
The table summarizes the three settings at 128 mean active tokens (IN100 = ImageNet-100; IN1k = ImageNet-1k).
| Tokenizer training | Evaluation | LPIPS KARL† ↓ | LPIPS GIST ↓ | DreamSim KARL† ↓ | DreamSim GIST ↓ |
|---|---|---|---|---|---|
| IN100 | IN100 | 0.296 | 0.231 | 0.278 | 0.200 |
| IN100 | IN1k | 0.297 | 0.232 | 0.298 | 0.221 |
| IN100 + 20 IN1k epochs | IN1k | 0.294 | 0.215 | 0.292 | 0.194 |
These results show that GIST’s reconstruction advantage at reduced token counts extends to ImageNet-1k, both without additional training and after matched continuation.
Response to Reviewer 2z8g (Part 3/3)
Initial generation results show benefits and trade-offs
[…] As discovered by previous work, better reconstruction performance doesn't necessarily translates to better generation performance. Therefore the lack of results on generative models significantly undermine the authors' contribution of the new proposed tokenizer.
We agree that improved reconstruction does not establish improved generation, and we do not interpret our results as evidence of better generative performance. Our contribution concerns whether semantic fidelity can serve as an explicit control signal for adaptive token allocation and preserve semantic content under compression. Our controlled token–fidelity comparisons and semantic probing directly evaluate this contribution.
Adaptive token allocation and the token–fidelity trade-off are established research objectives that have been studied independently of downstream generation. Closely related examples include ALIT (ICLR 2025), ElasticTok (ICLR 2025), KARL (NeurIPS 2025), and InfoTok (ICLR 2026 oral), which evaluate allocation, reconstruction, or representation properties without downstream image/video-generation experiments. This scope also extends to quantization research: the Rotation Trick (ICLR 2025) and SimVQ (ICCV 2025) explicitly distinguish improvements to tokenizer properties from demonstrated generation benefits. As SimVQ states, “evaluating VQ models primarily through downstream generation tasks may not provide the most insightful assessment of their fundamental properties.” Our contribution similarly concerns semantic control of adaptive allocation, which our token–fidelity comparisons and semantic probing evaluate directly.
Nevertheless, to investigate downstream applicability, we have also added a controlled class-conditional generation study in the appendix, “Generation Quality across Active Token Counts”, using matched 91.6M-parameter MaskGIT-style generators trained for 50,000 updates each on ImageNet-100. The generators predict both code values and activity from class labels alone. Our initial generation study provides encouraging evidence of downstream benefits. Over the shared token range, the interpolated curves show lower gFID without classifier-free guidance and higher class consistency and precision, but lower recall, at both evaluated guidance settings. For example, without guidance, the measured points near 37 mean active tokens (36.67 for GIST; 36.92 for KARL†) give gFID of 69.03 versus 71.25 and class accuracy of 15.99% versus 14.13%, respectively. At these points, precision is higher (0.488 versus 0.430), but recall is lower (0.422 versus 0.468). At the tested guidance scale , GIST achieves better gFID than KARL† at some token counts and worse gFID at others. We present these findings as an initial study of downstream use, while our central claim remains the semantic control and token-fidelity benefits evaluated directly in the paper.
We would appreciate your feedback on whether our responses address your main concerns, and would be happy to clarify any remaining points.
Response to Reviewer 2z8g (Part 1/3)
We thank the reviewer for the detailed feedback and for highlighting the value of semantic protection when pixel-level targets alone are insufficient for adaptive tokenization. Below, we clarify GIST’s contribution relative to prior uses of semantic information in tokenizers, summarize the broader tokenizer comparisons, and present additional ImageNet-1k and downstream generation experiments.
Semantic fidelity targets guide adaptive token allocation
We agree that incorporating semantic information from pretrained vision models is established practice in image tokenization, as discussed in our related work. We believe our contribution concerns a fundamentally different use of this information: using semantic fidelity to control adaptive token allocation. GigaTok and ReTok use semantic alignment to shape the representations learned during training; VFMTok builds its representation on foundation-model features and semantic reconstruction. GIST instead lets the user specify a desired semantic fidelity, which guides the tokenizer’s decision about how many tokens to retain and what information those tokens preserve. Methodologically, we also operate differently: GIST introduces no DINO feature-alignment loss; rather, a frozen DINO model measures the semantic error used for this conditioning. To make this distinction easier to locate, we will also state it explicitly next to the training objective in Section 3.2.
The distinction therefore goes beyond introducing another semantic loss objective: semantic fidelity becomes a controllable criterion for token allocation. The cited methods do not provide this control. ReTok supports variable-length representations, but its semantic regularization depends on the manually selected sequence length; GIST uses the requested semantic fidelity to guide the allocation itself. Our contribution is this use of semantic fidelity for adaptive allocation, together with the experimental characterization of its behavior and benefits.