acceptodds
Withdrawn from ICLR 2027

MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Efficient visual tokenization requires balancing representation compactness with the preservation of fine-grained image content. Conventional grid-based tokenizers allocate tokens uniformly across spatial locations despite non-uniform visual complexity, whereas compact 1D tokenizers aggregate global information but can struggle to preserve local details. We introduce MixTok, a hybrid visual tokenizer that combines a compact 1D global representation with sparse 2D tokens for local refinement. A content-adaptive router ranks candidate locations by refinement priority and determines the number of 2D tokens for each image under a target average token budget. At the highest-scoring locations, 2D token features replace the spatial features recovered from the 1D representation, and a shared decoder reconstructs the image from the resulting mixed features. On ImageNet , MixTok achieves highly competitive reconstruction and generation performance, with an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 content tokens per image.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
visual tokenization, masked image generation, adaptive token allocation, hybrid representations, image reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_G9NBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces MixTok for image tokenization. It keeps a short 1D token sequence as a global base and adds 2D tokens only at selected locations for local refinement. A grid scorer ranks spatial positions by refinement priority, and a top-p budget controller decides how many 2D tokens each image receives under a target average budget. At the chosen locations, 2D features replace the spatial features expanded from the 1D tokens, and a shared decoder reconstructs the image from the mixed grid. For class-conditional generation, a two-stage MaskGIT first produces the 1D tokens, which then determine where and how many 2D tokens to generate. Experiments on ImageNet shows that MixTok uses about 128 content tokens to reach an rFID of 1.195 and a PSNR of 19.51 dB, demonstrating the effectivess of the proposed method.

Critical strengths
  • The core idea is simple and intuitive: use compact 1D tokens for global structure and add sparse 2D tokens only where local refinement is needed. This gives a clear way to combine the strengths of 1D and 2D tokenizers.

  • The empirical results are strong at the tested setting. MixTok achieves competitive reconstruction quality at about 128 content tokens and obtains the best gFID among the reported generation baselines in Table 3.

Critical weaknesses
  • The practical benefit is not fully convincing. MixTok stores fewer content tokens, but reconstruction still uses a dense 2D encoder and a dense 16×16 decoder grid, while generation still runs on the full spatial sequence. As a result, the method does not clearly translate token reduction into compute reduction, and its latency/FLOPs are still higher than some baselines such as LPD-L-256.

  • The experimental scope is limited. All main results are on ImageNet 256×256 with a narrow token-budget range, which is not enough to validate the claim that sparse 2D refinement preserves fine details. Higher-resolution results or detail-heavy scenarios such as text rendering, documents, or thin structures would make the evidence more convincing.

  • The related-work comparison is incomplete. Keep-or-Drop (Adaptive Tokenizer for Compact Video Representation) appears closely related because it also studies which tokens to retain under a compact budget, but it is not discussed or compared. The paper also cites spatial adaptive tokenizers such as DQ-VAE and GPSToken without evaluating against them, making it hard to isolate the benefit of MixTok’s hybrid design.

Questions for authors
  1. Since MixTok still uses dense 2D encoding, dense decoding, and a full spatial sequence in the second-stage generator, could the authors clarify the main practical use case where the reduced number of stored content tokens is more important than end-to-end compute or latency?

  2. Could the authors provide a more detailed compute breakdown for tokenization and generation, including the costs of the 1D branch, 2D branch, grid scorer, budget controller, and shared decoder?

  3. How well does MixTok generalize beyond ImageNet 256×256? In particular, can the authors provide results at higher resolutions or on detail-sensitive data such as rendered text, documents, thin structures, or small objects?

  4. How does MixTok compare with adaptive or sparse tokenization methods that explicitly decide which tokens to keep or drop, such as Keep-or-Drop, DQ-VAE, or GPSToken, under matched token budgets and compute budgets?

Reviewer_MhQ4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper approaches the problem that 2D tokenizer’s latents are not proportional in their size wrt the inherent image content but can model local details well, while 1D tokenizers are compact but may not model local details well. To that end they propose a hybrid approach, where 1D image representations are enhanced with a subset of 2D tokenizer tokens that “fill in” the more local detail that the 1D tokenizer missed. For that, a scorer predicts the relevant locations, the features are mixed, and a decoder reconstructs the image. For generation then a 1D MaskGIT generates the global tokens, which then is fed to the scorer that guides a 2D MaskGIT model where to refine.

Critical strengths

The idea is relatively simple to understand and nicely modular. I find the overall motivation of a hybrid representation that is both global and local where it matters interesting and worthwhile to investigate.

Critical weaknesses

I think one of the main weaknesses is that it’s pretty unclear whether the gains come from the proposed 1D + 2D mixing, or if it’s more from the fine-tuning recipe and the pre-trained Halton MaskGIT generator. The llamagen decoder and 2D encoder are finetuned with a DINOv2 / CLIP loss and MixTok with only 96 of llamagen’s tokens beats llamagen with all 256 tokens, suggesting it’s the decoder that improved. A K=0 and K=256 result would help here. I also think that the entire method is hard to evaluate if there are barely any qualitative examples in the paper, and the ones that are there do not look very convincing given the high complexity of the proposed method. Evaluations are almost entirely on FID, which is brittle and gameable, especially when training with a feature loss and pretrained discriminator. One last major point about careful ablations: I feel that most effort is put into ablating just some modules of the architecture, but not major design decisions, and also controlled baselines and experiments are missing. For example, there is no controlled ablations on the number of tokens, and the comparisons are not train nor inference compute matched.

Questions for authors
  • What are MixTok’s rFID at K=0 and K=256?
  • What is LlamaGen’s rFID after fine-tuning with the full loss recipe proposed?
  • It would be good to see full rate-distortion curves against variable length tokenizers like FlexTok, DetailFlow, and One-D-Piece.
Reviewer_1iMcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MixTok addresses the trade-off between compact visual representations and preservation of local image details. It combines pretrained global 1D tokens with selected 2D tokens through spatial feature replacement, using a learned scorer and top-p controller to choose refinement locations and image-dependent budgets. A distilled scorer and embedding adapter enable two-stage class-conditional MaskGIT generation. On ImageNet 256 × 256, the method reports rFID 1.1950 and PSNR 19.51 dB at 128.11 average content tokens, and gFID 1.9511 ± 0.0072 at 130.66 tokens with 8 + 8 sampling rounds.

Critical strengths
  1. The hybrid representation delivers a useful measured generation trade-off: Table 3 reports gFID 1.9511 at 270.1 ms versus TiTok-S-128’s 1.9994 at 1018.4 ms; Appendix A.8 includes both generation stages and RGB decoding in latency, and Table 9 confirms consistency across four sampling seeds.
  2. The reconstruction ablations meaningfully separate spatial selection from budget allocation: Table 4 holds per-image budgets constant and improves rFID from 1.5460 with random selection to 1.1950 with learned selection, while Table 5 preserves the budget histogram in its shuffled control.
  3. The hybrid design transfers beyond one tokenizer pair: Table 2 improves ALIT–VQGAN reconstruction from rFID 2.9453 to 2.4720 at the same 128-content-token budget, complementing the TiTok–LlamaGen configuration in Section 5.1.
  4. The implementation and accounting are explicit: Appendices A.1–A.4 specify modules, training schedules, losses, and sampling, while Eq. (22) and Table 10 include the spatial mask in bitrate accounting rather than equating content-token count with storage cost.
Critical weaknesses
  1. Generation experiments do not isolate the adaptive router’s contribution. Tables 4–5 evaluate reconstruction; Table 9 compares a 32-token base against a larger representation with an additional generator and eight additional rounds. This establishes the value of the complete refinement stage, but leaves unclear whether learned locations and variable budgets improve generation over simpler refinement. Compare fixed-K learned routing and matched-budget random/uniform routing under matched training and sampling settings, reporting gFID and latency.
  2. The reconstruction efficiency trade-off is insufficiently characterized across budgets. Tables 1 and 10 concentrate on approximately 128 tokens; Table 10 gives MixTok 0.03030 bpp versus DetailFlow’s 0.02539 bpp, with rFID 1.1950 versus 1.1370 and PSNR 19.51 versus 19.46. Thus, the evidence supports a particular trade-off rather than a superior rate–quality frontier. Report several budgets and global/local splits, with position-inclusive bitrate; Section 6 appropriately acknowledges the narrow evaluation range.
  3. The incremental benefit of variable counts is small and its training robustness is unresolved. Table 5 improves rFID by 0.0102 and PSNR by 0.0483 dB over fixed K; the reported deviations cover shuffled assignments, not independent training runs. Table 9 likewise measures sampling variability for one checkpoint, as Appendix A.4 explicitly states. Independent training repeats, or appropriately qualified conclusions, would clarify the reliability of this incremental component.
Questions for authors

See the weaknesses.

Reviewer_HMLEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MixTok, a hybrid visual tokenizer that combines compact global 1D tokens with sparse 2D tokens for local refinement. The method uses a learned grid scorer to identify regions that benefit most from additional details, and a top-p budget controller to dynamically allocate 2D tokens based on image content. The selected 2D features replace the corresponding 1D-derived features before reconstruction through a shared decoder.

The authors also extend this framework to image generation using a two-stage MaskGIT pipeline. Experiments on ImageNet 256×256 show competitive reconstruction and generation performance, achieving an rFID of 1.195 with around 128 tokens and a gFID of 1.95 with around 131 tokens. Additional ablation studies and experiments with different tokenizer backbones are provided to validate the proposed design

Critical strengths

The idea of using compact 1D tokens for global information and selectively adding 2D tokens for local details is reasonable and effective. MixTok achieves competitive reconstruction quality with a relatively small token budget. The generation results are also strong, with a gFID of 1.95, showing that this hybrid representation can work well for both reconstruction and generation.

The authors compare the learned grid scorer with several alternative selection strategies, including random, uniform, and gradient-entropy-based selection. The results show that the learned scorer consistently performs better. The comparison between fixed, dynamic, and shuffled token budgets also provides evidence that content-aware allocation is useful. The experiments with the ALIT backbone further support the transferability of the method.

Critical weaknesses

The method mainly combines existing 1D and 2D tokenizers with a learned scorer and top-p selection. Most of these components are based on existing techniques, and the main contribution is how they are combined. While the overall design is reasonable, the technical novelty seems somewhat limited.

The improvement from dynamic token allocation over fixed allocation is relatively small. In Table 5, rFID only improves from 1.2052 to 1.1950, despite introducing an additional budget controller. It is unclear whether this improvement is significant enough to justify the added complexity, especially without more comparisons across different token budgets.

Although MixTok reduces the number of retained tokens, reconstruction still requires dense 2D encoding, and the second-stage generator processes the full spatial grid. This means fewer tokens do not necessarily translate into proportional computational savings. More detailed efficiency analysis would help demonstrate the practical benefits of the proposed sparse refinement strategy.

Questions for authors

Since reconstruction still requires dense 2D encoding and the second-stage generator processes the full spatial grid, could the authors clarify how much computational cost is actually saved by the sparse refinement strategy? A breakdown of the runtime and computational cost of different components would help clarify this.

The improvement from dynamic token allocation over fixed allocation appears relatively small in Table 5. Could the authors provide results across different token budgets to demonstrate whether the adaptive allocation consistently offers meaningful advantages? This would help clarify the contribution of the budget controller beyond the hybrid representation itself.

Since the tokenizers use different codebook sizes and MixTok requires additional spatial position information, how does the method compare with existing approaches under matched bitrate or computational budgets rather than matched token counts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.