acceptodds
Under review as a conference paper at ICLR 2027

MoETok: Sparse Expert Routing for 1D Visual Tokenizers

Abstract

Modern visual tokenizers aim to preserve fine-grained visual details while learning high-level semantic representations for downstream generation and understanding. Query-based 1D visual tokenizers encode images into flexible latent sequences, yet how individual learnable queries gather and encode image information remains poorly understood. We reveal that these queries spontaneously develop complementary roles under semantic supervision: a small set of Content queries captures high-level semantic information, while a larger set of Detail queries preserves fine-grained visual details for reconstruction. To adapt token processing to these heterogeneous roles, we introduce MoETok, a 1D visual tokenizer that routes tokens through sparsely activated experts. Jointly optimized with reconstruction and semantic objectives, the router learns role-specialized processing paths adapted to each token's representation. Across model scales and token budgets, MoETok consistently improves reconstruction, semantic representation, and autoregressive generation over dense tokenizers under matched supervision and active computation. On ImageNet, it combines strong reconstruction and semantic representation with state-of-the-art class-conditional generation within the 1D tokenizer family. We further validate MoETok at scale in a unified autoregressive model jointly trained for understanding and generation with a next-token cross-entropy objective. The resulting model combines competitive visual understanding with state-of-the-art generation among unified discrete models, matching or surpassing diffusion-based models such as BAGEL and Qwen-Image on benchmarks including DPG-Bench (87.61), GenEval2 (40.04 GM), and WISE (0.49).

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual Tokenization, 1D Visual Tokenizers, Mixture of Experts, Sparse Routing, Discrete Representation Learning, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qKR7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies complementary content and detail roles among queries in semantically supervised 1D visual tokenizers and proposes MoETok, which introduces sparse expert routing. Experiments show improvements in reconstruction, semantic representation, and generation across model scales and token budgets, alongside applicability to unified multimodal modeling.

Critical strengths
  1. Extensive experiments demonstrate that MoETok improves reconstruction, representation, and generation across model scales and token budgets, supporting the effectiveness of the method.
Critical weaknesses
  1. Methodological novelty is limited. MoETok mainly applies standard sparse MoE to 1D visual tokenizers without introducing a new routing mechanism or expert architecture. The query-role analysis is informative but does not clearly translate into a design beyond conventional MoE.
Questions for authors
  1. With expert capacity matched, how much does learned routing improve over fixed random routing?
2 comments
AuthorsOpenReview ↗

Author Response: Research Contributions and the Value of Learned Routing

Thank you for recognizing our experiments across scales and token budgets and for suggesting the fixed-random-routing comparison. We appreciate the constructive discussion. Below, we clarify how the analysis motivates our design and present the additional evidence on learned routing.

1. Standard MoE components limit methodological novelty, and the analysis-to-design connection is unclear.

We respectfully disagree that “mainly applies standard sparse MoE” captures the full contribution. We present the first systematic cross-family analysis of spontaneous query heterogeneity in 1D visual tokenizers. Figure 1 shows recurring differentiation across architectures and training methods; masking and subset probes directly connect these behaviors to semantic and localized reconstruction preferences (Figure 2/Table 1). This motivates a testable hypothesis: task-learned transformations can better process heterogeneous queries, with unchanged supervision. In Table 2, applying MoE directly to a patch-based 2D tokenizer yields substantially smaller representation gains than in query-based 1D tokenization (LP +2.43 vs. +11.50 points, each against its matched Dense baseline). Figure 5 extends the quality–compute evidence across 12 configurations, and revised Table 3 separates capacity from learned expert selection. Full ImageNet and unified-model evaluations establish downstream value. The study also identifies tokenizer-specific choices: enforced balance worsens joint quality, while decoder routing trades higher LP for worse reconstruction/generation (original Table B.1; Appendix C.1/Table C.1). The contribution thus connects a cross-family finding, analysis-led design and controlled validation, rather than relying on component novelty alone.


2. How much does learned routing improve over fixed random routing at matched expert capacity?

Learning expert selection improves reconstruction, linear separability and generation at the same capacity and computation, with similar zero-shot accuracy. Following your suggestion, we randomly initialize and freeze the router while training the remaining network, matching expert count, width, Top-k and active/total capacity; assignments still depend on token inputs. Revised §4.2/Table 3 includes the pair reproduced below (199M/313M active/total encoder parameters; 155 encoder GFLOPs; 641 MiB peak FP16 forward memory at batch 1; shared 303M decoder; unguided gFID). Learning the router improves LP by 3.74 points and lowers rFID/gFID by about 0.144/1.65, showing that task-learned paths use the same available expert capacity more effectively.

RoutingrFID↓Linear probe↑Zero-shot↑gFID↓
Fixed random1.51156.4865.1111.24
Learned1.36860.2265.039.59
AuthorsOpenReview ↗

Thank You and Follow-up on Our Response

Thank you for suggesting the fixed-random-routing comparison. It helped us clarify the benefit of learned routing beyond expert capacity and strengthened our analysis (revised Table 3). Could you share your thoughts on the added results and whether they address your concerns? We would be happy to discuss any remaining questions.

Reviewer_GCziOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper first analyzes learnable queries in 1D visual tokenizers and reports that, under semantic supervision, they split into a small group of "Content" queries (image-adaptive attention, strong linear-probe contribution) and a larger group of "Detail" queries (fixed spatial attention, needed for reconstruction). Based on this, the authors propose MoETok, which replaces the encoder FFNs with sparse MoE layers so that tokens can be routed to different experts. MoETok is evaluated on ImageNet reconstruction/representation/generation and is also used as the visual tokenizer of a unified autoregressive model for understanding and generation.

Critical strengths
  1. The query-role analysis is the most interesting part. Across many 1D tokenizers, semantic supervision consistently produces a small set of image-adaptive Content queries, supported by interventions (Table 1, Fig. 2) and cross-dataset checks.

  2. The method is simple and the ablations are fairly complete. MoETok consistently beats the Dense baseline under matched active compute (Fig. 5), though the source of the gains is not fully clear (see W3).

Critical weaknesses
  1. No qualitative results. Generation is a main claim, but no generated images are shown for either ImageNet or the unified model. Side-by-side comparisons with other tokenizers / unified models are needed.

  2. The VqV_q threshold (Vq>0.5V_q > 0.5) is not justified, and the Content count (2 vs. 20) may depend on it. Since VqV_q is a ratio, a query whose mean attention is close to uniform gets VqV_q near 1 even with small cross-image variation. Please show results under other thresholds.

  3. MoETok matches Dense in active but not total parameters (845M vs. 505M). Table 2 and C.2 help, but neither isolates learned routing from capacity. A total-matched Dense or random-routing MoE baseline is needed.

  4. All experiments are at 256×256256 \times 256. It is unclear whether the role split and the expert / Top-kk choices still hold at higher resolution.

Questions for authors

See Weaknesses. Also: Top-1 routing in layers 1-2 is nearly collapsed (Neff≈1N_{\mathrm{eff}} \approx 1, Fig. B.4), and loss-free balancing hurts (Table B.1). Have other balancing strategies been tried, or is this concentration intended?

11 comments
AuthorsOpenReview ↗

Author Response: Query Analyses, Controlled Comparisons, and Additional Experiments

Thank you for your thoughtful feedback. We welcome this discussion and address your points below.

1. Generation claims need qualitative ImageNet and unified-model examples, including side-by-side comparisons.

Appendix D adds ImageNet samples (MoETok/LlamaGen-XL, 775M, 300 epochs; D.1/Figure D.1) and same-prompt MoETok/BAGEL/Qwen-Image GenEval2 outputs (D.2/Figure D.2), illustrating appearance, counts and spatial relations. Both panels and captions: anonymous PDF 5: generation examples, click to view.


2. A near-uniform mean attention can produce high Vq despite small absolute variation.

We check unnormalized variation: high-Vq Content queries exhibit measurable per-patch attention RMS across images (Dense: 0.73–0.77 times the uniform weight; MoETok: 0.59–1.08; common unit 1/256; anonymous image 7: absolute variation, click to view). Their functional interpretation has direct support from Figure 2/Table 1, beyond the relative statistic. The absolute statistics supplement the response.


3. The 0.5 threshold and the resulting Content counts may be sensitive to the chosen cutoff.

Grouping reveals cross-family query preferences, not precise binary labels. 0.5 splits Figure 1's clusters. We repeat the same evaluations as in Table 1, varying only the grouping threshold (anonymous image 2: thresholds (click)); the contrast persists at 0.5–0.7. Self-attention allows shared semantics, yet one Dense Content query nearly matches all queries (68.88% vs. 69.80% LP), versus 39.37% for one Detail query. Figure 2 shows object-following Content attention and spatially stable Detail attention matching local reconstruction effects (anonymous image 10: Figure 2 (click)). Both support query preferences.


4. Active-parameter matching does not separate routing benefits from total capacity.

Original §4.1/Appendix A.1 matched backbone, decoder, data, supervision, recipe and active FFN capacity. Following your suggestion, revised §4.2/Table 3 adds equal-total Dense. MoETok improves reconstruction, LP and generation with 43% fewer encoder GFLOPs than equal-total Dense, while ZS is similar. The comparison below shares a 303M decoder and unguided gFID (encoder GFLOPs: 155/155/271; peak MiB: 421/641/658, row order; FP16 forward, batch 1). It distinguishes the benefit from simply increasing dense capacity.

EncoderActive/total MrFID↓LP↑ZS↑gFID↓
Dense, equal active199/1991.52248.7259.1710.35
MoETok199/3131.36860.2265.039.59
Dense, equal total313/3131.44856.3664.9710.26

5. The role split and expert/Top-k choices have not been tested at higher resolution.

At 512×512, we use Table 2's Base encoder/Large decoder, 256 tokens, E4/T2 and 20-epoch ImageNet recipe, changing only resolution. MoETok improves reconstruction, representation and generation without retuning at matched encoder computation (gFID unguided). Attention-variance analysis shows similar query differentiation at 512×512 (anonymous image 11: 256/512 analysis (click)).

TokenizerEncoder GFLOPsDecoder GFLOPsrFID↓LP↑ZS↑gFID↓
Dense4259371.646750.45263.9710.49
MoETok4259371.495958.69668.339.24

6. Early-layer expert use is nearly collapsed; is this intended, and have other balancing strategies been tested?

The concentration is learned, not imposed: Top-1 N_eff summarizes first choices while Top-2 still activates two experts per token. We additionally test Switch's auxiliary loss (Fedus et al., 2022, weight 0.01), alongside the original loss-free comparison (Table B.1; revised Appendix B.3; anonymous image 9: expert utilization, click to view). More even expert usage does not improve joint quality in this 1D tokenizer comparison: both strategies increase early utilization but worsen all four metrics. Full-depth routing improves semantics over late-only routing, so concentrated early layers still contribute. Switch is added response evidence; tokenizer-specific balancing merits study.

BalancingrFID↓LP↑ZS↑gFID↓
None1.36860.2265.039.59
Loss-free1.49155.3963.2912.65
Switch (0.01)1.44559.1663.8410.73

512 × 512 is not high resolution. Furthermore, I have reservations about how well this performs on small faces and text. Finally, I also have doubts regarding the novelty of the paper.

AuthorsOpenReview ↗

Follow-up concerns, additional evidence, and rating clarification

Thank you for following up.

  1. Additional concerns and the revised assessment.

Our first response addressed your original concerns: qualitative examples, threshold and absolute-variation checks, a total-parameter-matched Dense baseline, higher-resolution results, and balancing strategies. The 512×512 test answered your request for a higher resolution, with the other settings unchanged.

What remains unclear is why the score was lowered from 2 (Weak rejection) to 1 (Clear rejection). Your follow-up raises additional concerns but does not explain what was unsatisfactory in our first response. Novelty is a major evaluation criterion, yet it was not questioned until after we had responded. We welcome further questions and respectfully ask what new evidence or unresolved issue led to this change. We respect the time and expertise you volunteer as a reviewer.


  1. Additional concern: “512 × 512 is not high resolution.”

Your original request was for a higher resolution. Our controlled increase from 256×256 to 512×512 directly addresses that request, with the other settings unchanged. The configurations in our main comparison (Table 6) are:

ConfigurationTokensOutput / evaluation size
TiTok32256×256
Semanticist32256×256
FlexTok32256×256
DetailFlow128256×256
VFMTok256336×336 / 256×256
ReTok256256×256
GigaTok256256×256
MoETok256256×256

Current research on compact 1D tokenizers focuses on representation quality and tokenizer design. MoETok advances this goal, achieving the lowest gFID (1.39) among these configurations. At 512×512, all four metrics improve over Dense at matched computation, and similar query differentiation remains (Appendix C.2, Figure C.1, Table C.4):

TokenizerEncoder GFLOPsDecoder GFLOPsrFID ↓LP ↑ZS ↑gFID ↓
Dense4259371.646750.45263.9710.49
MoETok4259371.495958.69668.339.24

These results support transfer to a higher resolution; calling 512×512 “not high resolution” does not invalidate them. Extending to still higher resolutions calls for designs tailored to longer spatial sequences and their quality–compute tradeoffs, a future direction discussed in Appendix E.


  1. Additional concern: “Furthermore, I have reservations about how well this performs on small faces and text.”

To test text and face reconstruction, we ran TokBench (Wu et al., 2026) [1].

Tokenizer1−NED (%) ↑FaceSim ↑
MoETok19.780.1803
Dense (GigaTok; Xiong et al., 2025 [2])17.990.1673
ReTok-S-B (Fu et al., 2026 [3])17.780.1485
TiTok B-L VQ128 (Yu et al., 2024 [4])13.140.1309
DetailFlow-16 (Liu et al., 2025 [5])11.960.1265
FlexTok (Bachmann et al., 2025 [6])11.570.1463

MoETok improves both text preservation and facial similarity over Dense.


  1. Additional concern: “Finally, I also have doubts regarding the novelty of the paper.”

In your initial review, you wrote, “The query-role analysis is the most interesting part,” and described the ablations as “fairly complete.” You did not raise a novelty concern then.

We provide the first systematic cross-family analysis of spontaneous query heterogeneity in 1D visual tokenizers (Figures 1–2, Table 1). We then test whether token-dependent processing can improve these heterogeneous representations. Simply adding MoE to the patch-based 2D tokenizer yields limited gains, whereas the query-based design benefits substantially (Table 2: Linear Probe +2.43 vs. +11.50 points, each against its own matched Dense baseline). We test routing depth and expert configurations, compare 12 configurations in Figure 5, and control for capacity and routing in revised Table 3.

References

[1] Wu, Junfeng, et al. (2026). TokBench: Evaluating Your Visual Tokenizer before Visual Generation. Proceedings of the 34th ACM International Conference on Multimedia.

[2] Xiong, Tianwei, et al. (2025). GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation. ICCV.

[3] Fu, Zixuan, et al. (2026). Improving Flexible Image Tokenizers for Autoregressive Image Generation. NeurIPS.

[4] Yu, Qihang, et al. (2024). An Image is Worth 32 Tokens for Reconstruction and Generation. Advances in Neural Information Processing Systems, 37, 128940–128966.

[5] Liu, Yiheng, et al. (2025). DetailFlow: 1D Coarse-to-Fine Autoregressive Image Generation via Next-Detail Prediction. arXiv:2505.21473.

[6] Bachmann, Roman, et al. (2025). FlexTok: Resampling Images into 1D Token Sequences of Flexible Length. Forty-second International Conference on Machine Learning.

Reviewer_PFhPOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This work proposes a visual tokenization strategy primarily for image generation based on a MoE structure. The work starts by the observation that under a query-based token compression/distillation mechanism, a certain subset of tokens have a tendency to attend to multiple regions of the input images, termed as "content tokens", while another subset tends to exhibit more localized attention patterns to certain spatial regions, termed as "detail tokens". Following up from this, the authors propose to replace the FFN blocks in existing visual tokenizers with MoE blocks, where different experts could be activated for different tokens. The authors then attempt to demonstrate the usefulness of the proposed mechanism under image generation benchmarks along with some experiments on unified any-in-any-out LVLMs.

Critical strengths

The work has the following two primary strengths:

  • Experimental Strength w.r.t. Ablations and Some Image Generation Settings

    • The work has decent ablations on the placement of the MoE blocks w.r.t. the visual tokenizer layer indices (Table 3) and also on different number of total/active experts (Table 4), which mostly seem to support the primary set-up presented in the work.
    • Although not being fully apples-to-apples (MoETok has substantially more total parameters compared to the baseline GigaTok, only matching in active parameters which isn't a 1:1 comparison), there is at least some gains with the proposed approach over existing works (e.g. w.r.t. GigaTok on Table 5, +0.06 rFID and +0.38 gFID).
    • Likewise, although not fully apples-to-apples (differing in backbones, training data, post-training strategies and many hyperparameters), the proposed approach seems to have a basic working proof-of-concept adaptation to unified any-in-any-out LVLMs.
  • Rigor in Scientific Presentation with Eye-Appealing Figures

    • Although many are under-specified, the work has no shortage of illustrative figures, all of which appear to be of high quality and high effort.
    • To exemplify, Figure 1 qualitatively supports the role differentiation across tokens, while Figure 2 shows how the detail tokens tend to attend to the same regions despite the particular visual input.
Critical weaknesses

The work has the following primary weaknesses:

  • W1: Problems with the Motivation and Narrative
    • The biggest issue I can see with these parts is that these sections lack stand-alone precision. There are many eye-appealing visualizations all of which seem to support the narrative of the work, sure, but precisely which model, on which data and under which settings was used to generate those is not detailed in the main text (e.g. for Figure 2 and Table 1). Although I understand that the space limitations might have caused this, as a reader, it is very hard to fully understand the substance behind the visualizations in a clear manner. Accordingly, any revision of the work should clearly and concretely identify the core settings behind the visualizations more clearly. As it stands, Section 3 is very hard to follow.
    • Another narrative issue to me stems from going from the motivation on the dichotomy of token roles vs the MoE structure. The statement "These findings motivate token-dependent parameter allocation: each token selects a subset of experts from a larger parameter pool, with active computation comparable to dense processing." reads a lot more like a self-fulfilling prophecy: There could be diverging roles within tokens, which is an "emergent" property as the authors term, but how does this necessitate directing the computation for each subset? Is there a fundamental gap caused by the lack of computation reserved for each sub-group of tokens? There are some linear probing/rFID numbers on Table 1 to demonstrate how MoETok improves over the Dense baseline, though the numbers could very well be confounded since (i) it is not precise what the difference between Dense vs MoeTok is and (ii) MoETok might have more number of total parameters (which from a latency perspective doesn't add much but has other computations overhead, e.g. memory) or has gone through a different training set-up. These points should ideally be clarified to tie the motivation to the proposed method.

  • W2: Problems with the Attention Analyses and Section 3
    • The work cites several works for the attention analyses it presents, which is appreciated. However, this doesn't justify certain design choices taken by the authors to motivate certain design choices that were made.
    • Notably, the authors state that they average the VqV_q values across layers. However, averaging across depth in such a way could be problematic, given existing works in the literature note that certain layers in vision transformers are subject to having outlier representations, e.g. register/sink tokens emerging only after a certain index [A, B]. Ideally, to justify such aggregation, the authors should present how these patterns across different layers vary (in the form of a matrix of VqV_q values across tokens/layers) and whether there are any outlier layers. If the behavior across the layers isn't fully consistent, then the utilized aggregation operation could be biasing the observations made in the work.
    • Finally, the work claims that content tokens capture "high-level semantic information", which is primarily demonstrated by the indirect attention patterns of such tokens. Although I acknowledge the common use of attention patterns for such purposes in the literature, I believe that the work's analyses could be strengthened by principled frequency analyses, e.g. by analyzing frequency-band or spectra captured by each token similar to the analysis conducted in existing works [C, D]. This is more of a nice-to-have and could demonstrate to what extent the content tokens are capturing the "high-level semantic information" or the low-frequency layout of the underlying image features.

  • W3: Problems with the Experimental Analysis
    • The most important issue with the experiments is that many of them aren't apples-to-apples: In the primary result table where MoETok is compared against the baselines, MoETok's only matching setting with them is its number of tokens. Although I acknowledge that MoETok has impressive gains over GigaTok-XL-XXL, it seems to bring very modest improvements over the primary baseline of GigaTok on major image generation measures, e.g. +0.06 rFID and +0.38. This is despite MoETok having substantially more trainable parameters compared to GigaTok (845M vs 505M), while only matching it at the active number of parameters.
    • This issue is also repeated across several different result tables, e.g. Tables 7, 8 and 9, where the rows on the table follow different backbones, different training data, different hyper parameter settings and more. Although I understand that it is simply infeasible to control for each of these settings, the tone of the narrative as it stands is over-emphasizing the gains of MoETok, whereas it should have been modestly stating that those tables are merely there as a proof-of-concept. Any other claim otherwise should ideally be substantiated with controlled and matched experiments.
    • Finally, the results under Table 6 on multimodal understanding have more of a heuristic value on demonstrating that MoETok does not entirely erase the understanding capabilities, rather than preserving them effectively. Many of the benchmarks presented in Table 6 with strong retention are known benchmarks which with very small number of visual tokens, it is possible to retain most or all of the performance, as shown in visual token compression works from the multimodal understanding literature [C, D, E]. For the benchmarks on Table 6 that aren't primarily reported in these works, for MMB and MMMU, MoETok has a very heavy negative delta against Qwen3.5-4B, −40-40 pts at MMMU and −14-14 pts at MMB, which further reduces the significance of results reported under Table 6.

  • W4: Missing Related Works from the Literature
    • As a final weakness, the work is overlooking many existing works from the literature on query-based (or 1D) visual tokenization for multimodal applications. Although I acknowledge the application domain difference between these works and MoETok, for good scientific practice, acknowledging these work is a nice-to-have.
    • These could be listed as Matryoshka Query Transformers [E] which is the predecessor of FlexTok with a nearly-identical mechanism although used for understanding and other related works from the vision-language understanding literature [D, F] which also follow from a role-based dichotomy of visual tokens akin to MoETok should at least be cited.
    • Finally, an existing arXiv work that came out in March 2026 [G] seems to have a similar promise with MoeTok, although importantly for understanding and not image generation.

[A] Jiang, N., Dravid, A., Efros, A., & Gandelsman, Y. (2026). Vision Transformers Don't Need Trained Registers. Advances in neural information processing systems, 38, 56557-56595.

[B] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024, May). Vision transformers need registers. In International conference on learning representations (Vol. 2024, pp. 2632-2652).

[C] Pan, Q., Dou, Z., & Li, S. (2026). Frequency-modulated visual restoration for Matryoshka large multimodal models. In Conference on Computer Vision and Pattern Recognition 2026.

[D] Kuzucu, S., Tonioni, A., Lup, V., Schiele, B., Tombari, F., & Naeem, M. F. (2026). PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding. arXiv preprint arXiv:2605.30126.

[E] Hu, W., Dou, Z. Y., Li, L. H., Kamath, A., Peng, N., & Chang, K. W. (2024). Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37, 50168-50188.

[F] Zhang, J., Guo, X., Cai, K., Lv, Q., Fan, Y., Chai, W., Wang, J., & Wang, K. (2026). Hybrid token compression for vision-language models. In Conference on Computer Vision and Pattern Recognition 2026.

[G] Li, Z., Li, Y., Fang, F., Takezoe, R., Bo, Z. H., Qian, C., ... & Long, K. (2026). QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression. arXiv preprint arXiv:2603.21232.

Questions for authors
  • Q1: Could the authors share their thoughts on the issues raised under W1: Problems with the Motivation and Narrative? I think a clarification and a revision of those sections could greatly improve the readability and presentation of the work.

  • Q2: Could the authors share how the patterns across different layers vary (in the form of a matrix of VqV_q values across tokens/layers), as raised under W2: Problems with the Attention Analyses and Section 3?

  • Q3: Could the authors also comment on why 0.50.5 was chosen as a threshold for VqV_q, and how do the linear probing accuracy and other material observations change with different thresholds? Sharing the distribution of content vs detail tokens w.r.t. different thresholds could make it easier for the readers to understand the underlying design choices better.

  • Q4: Could the authors present a similar frequency spectra analysis as mentioned under W2: Problems with the Attention Analyses and Section 3?

  • Q5: On Figure 1, the authors show that FlexTok also has a content vs detail token separation. As a nested query method, FlexTok naturally induces an order on its tokens, where they go from more general to granular towards the latter tokens. Do the observations of authors align with this natural expectancy from FlexTok? If so, to what extent?

  • Q6: Could the authors report results with more vision-centric benchmarks, such as ChartQA and DocVQA with the setting of Table 6 both with MoETok and with the base Qwen they adapt MoETok on top off?

  • Q7: Can the authors establish the asked connections between their works and previous works from the multimodal understanding literature? This is more of a nice-to-have.


Overall, I am willing to change my score contingent upon the authors' response. I am aware that I have raised quite a few issues, though I would not want this to discourage the authors from responding, I am open to discussion.

7 comments
AuthorsOpenReview ↗

Author Response (Part 1/2): Experimental Settings, Motivation, and Query Analyses

Part 1 of 2

Thank you for your thoughtful, detailed review and openness to discussion. We greatly appreciate your careful reading and welcome the opportunity to clarify our study. We reply in two parts: Part 1 addresses the settings, motivation and query analyses; Part 2 covers the remaining questions, experimental comparisons and related work.

1. Section 3 does not clearly identify the models, data and settings behind its figures.

Thank you for this helpful suggestion. The original paper provided these settings in §4.1/Appendix A.1; we now state them before the analysis in revised §3.1. All Dense–MoETok analyses use the same paired checkpoints (Figures 1, 2 and 4; Table 1). Decoder, supervision and training settings are identical; only encoder dense FFNs change to E4/T2 sparse expert layers at matched active computation. Figure 1 also includes other tokenizer families' released models.


2. Emergent query roles do not by themselves justify directing computation for each subset; is there a “fundamental gap” caused by the lack of reserved computation?

We appreciate your concern, but respectfully disagree with “self-fulfilling prophecy.” Our aim is to test a design hypothesis, not to establish a fundamental defect in dense FFNs. Figure 1 shows query heterogeneity emerging across existing dense 1D tokenizers; Figure 2/Table 1 connect it to semantic information and localized reconstruction. We therefore ask whether token-dependent parameter allocation can improve query-based tokenizer representations at matched active computation. The controlled tests show much larger representation gains for query-based than patch-based tokenization (Table 2: LP +11.50 vs. +2.43 points, each against its matched Dense). Reconstruction, semantic representation and generation improve jointly across 12 model configurations (Figure 5). In the trained MoETok, experts also learn token-role preferences in middle and final encoder layers (Figure 4), with a common Top-k and no role supervision. Revised §3.2 clarifies this motivation and testable hypothesis.


3. The Dense–MoETok differences and training settings are unclear, and more total parameters may introduce overhead, e.g. memory.

The original comparison matches active parameters and encoder computation (155 GFLOPs for both Dense and MoETok), keeping the backbone, decoder, supervision and training settings identical. This tests token-dependent parameter allocation at matched active computation. To address your concern about total parameters and memory, we additionally train equal-total Dense (revised §4.2/Table 3). At similar peak forward memory (641 vs. 658 MiB), MoETok improves reconstruction, linear probe accuracy and generation with 43% fewer encoder GFLOPs, while zero-shot accuracy is similar. Both quality and resource comparisons follow (shared 303M decoder; unguided gFID; FP16 forward, batch 1, 256×256).

EncoderActive/total MrFID↓LP↑ZS↑gFID↓
Dense, equal active199/1991.52248.7259.1710.35
MoETok199/3131.36860.2265.039.59
Dense, equal total313/3131.44856.3664.9710.26
EncoderGFLOPsPeak memory, MiB
Dense, equal active155421
MoETok155641
Dense, equal total271658

4. Averaging attention variance across depth may hide outlier layers and bias the observed grouping.

The Content subset is not determined by a single layer. The new 12×256 matrix retains query order and marks Content (Appendix B.1/Figure B.2; anonymous image 1: layer–query matrix, click to view). Omitting each layer preserves all original Content queries; mean label retention is 99.54%/100% (Dense/MoETok; additional response statistics).


5. The semantic interpretation relies on indirect attention evidence and would benefit from frequency analysis.

We appreciate your concern, but respectfully disagree that the semantic interpretation is “primarily demonstrated by the indirect attention patterns of such tokens.” The original manuscript directly tests semantic information through Figure 1's per-query Linear Probe and Table 1's subset probes, validating the functional interpretation of the attention patterns. Following your suggestion, we additionally analyze PARCEL-style spectra of query-attention-weighted final encoder features, summarized by power-weighted mean frequency (anonymous image 3: query spectra, click to view). Each model's lowest-frequency query is Content; MoETok's lowest nine are all Content. This complementary evidence is added in Appendix B.2/Figure B.5.

AuthorsOpenReview ↗

Author Response (Part 2/2): Thresholds, Baseline Comparisons, and Related Work

Part 2 of 2

Thank you again. We now turn to thresholds, FlexTok, experimental comparisons and related work.

6. The choice of 0.5 and the effect of other thresholds on group sizes and functions need clarification.

We aim to reveal query preferences across tokenizer families, not assign a precise binary label to every query. Figure 1 shows clusters; 0.5 provides a common midpoint without clustering or fitting. As suggested, we repeat the same evaluations as in Table 1, varying only the grouping threshold (anonymous image 2: thresholds (click)). At 0.5–0.7, Content subsets retain high LP accuracy and Detail subsets retain reconstruction quality. Self-attention permits shared semantics: one Dense Detail query reaches 39.37% LP, yet one Content query nearly matches the full sequence (68.88% vs. 69.80%). Figure 2 shows Content attention following objects across images and spatially stable Detail attention matching localized reconstruction effects (anonymous image 10: Figure 2 (click)). Quantitative tests and direct observations jointly support query preferences, motivating conditional processing.


7. The observed FlexTok groups should be related to its coarse-to-fine prefix order.

We find an overall prefix trend: mean retention probability and mean Vq have Pearson r = 0.88 (anonymous image 4: prefix exposure and Vq, click to view). Nested dropout retains earlier queries more often for decoding, encouraging broadly useful information near the start of the sequence. This connects the observed aggregate pattern to FlexTok's training mechanism.


8. The main result table is not “apples-to-apples”: only the token count matches, while MoETok has more total parameters.

Respectfully, “only matching setting with them is its number of tokens” does not apply to our reproduced Dense (GigaTok). This is a controlled 100-epoch baseline; “GigaTok” denotes its GigaTok-style supervision and training recipe. Backbone, data, supervision, training and active parameters match MoETok, as do LlamaGen-B settings. GigaTok-B-L/XL-XXL are official results. The equal-total control is given in clarification 3 (revised §4.2/Table 3). The full ImageNet comparison improves reconstruction, semantic representation and generation jointly.


9. Different unified-model backbones, data and training make tokenizer-specific attribution difficult.

Thank you for your feedback on our wording. The original Introduction distinguishes “controlled comparisons, ImageNet benchmarks, and large-scale multimodal modeling,” presenting unified modeling as further downstream validation. Revised §4.6 makes this distinction more explicit.


10. Large understanding gaps against Qwen call for paired ChartQA and DocVQA tests.

Thank you for suggesting paired ChartQA/DocVQA tests. The original Qwen references came from its technical report, whereas our new lmms-eval comparison uses identical images/padding at 256×256. Native dynamic-resolution preprocessing gives Qwen a larger visual-input budget (our counts: ~7.0× on MMMU, ~2.4× on MMBench); fixed input removes this major discrepancy, while training and continuous/discrete representations still differ. Under the more comparable input conditions, MoETok exceeds its Qwen3.5-4B base on six of nine understanding tasks while adding strong generation. MMMU/DocVQA retain gaps and motivate further improvements to fine-grained/text understanding. The paired results below provide system-level evidence of this downstream potential, not strict tokenizer attribution; they are supplied in the response.

Task↑Qwen3.5-4B (256)MoETok (256)
POPE85.085.5
MMB75.375.6
MME-P16091525
MMMU49.339.7
GQA55.760.0
VQAv275.476.7
SEED-Image73.274.2
ChartQA41.16043.360
DocVQA35.70326.363

11. Connections to prior query-based compression, complementary token paths and routed projectors are missing.

Thank you for these connections. Revised §2 now acknowledges prefix training (MQT), complementary visual paths (PARCEL/HTC-VLM), and query-guided projector routing (QMoP). Our focus is the first systematic cross-family analysis of naturally emerging query heterogeneity in 1D visual tokenizers, linked to semantic and reconstruction functions, and its use to guide conditional transformations inside the tokenizer. We test this hypothesis with unchanged task supervision and joint reconstruction, representation and generation evaluation. These additions clarify the relationship to understanding-oriented compression and routed projectors while preserving this distinct research contribution.

I would like to thank the authors for their kind and detailed response. As I stated in my initial review, I did raise an above average number of weaknesses. I am regardless pleased to see that the authors have provided a comprehensive list of updates. Accordingly, the following concerns I have raised were entirely or mostly addressed:

  • Narrative of Section 3: The authors have added a clear and concise paragraph describing the conditions that led to the figures and analyses, along with several key clarifications. Accordingly, I believe that the narrative issues around Section 3 have mostly been resolved.

  • Problems with Attention Analyses: The authors have, as asked for, provided the relevant matrix along with a frequency spectrum analyses. The new results all appear to support the paper's primary hypotheses and I do not see any immediate weaknesses around these.

  • Problems with the Experimental Analysis: Although some future clarity on the precise Qwen setting would be appreciated here (e.g. how would MoETok work with Qwen's native settings, or the new and upcoming LVLMs and their differing visual token processing strategies? I would encourage the authors to add at least some comments on these under a corresponding Related Work section, maybe in the Appendix if the spacing isn't sufficient), I can clearly see that the results provided by the authors under comparable settings demonstrate that MoETok is mostly on-par with the original Qwen model, except for DocVQA. Given the short time frame around the rebuttal period, I find it understandable that the default settings used in the work thus far may not be optimal for every single benchmark.

  • Missing Related Works from the Literature: The authors have acknowledged the prior works from the vision-language literature. My concerns on this end were never that a similar approach for the same domain have been realized, but rather methodological similarity for a different application domain.

I will be revising my score to a clear accept since the authors have adequately addressed a significant majority of the weaknesses I initially raised. I also went through the other reviewers and the author responses to them and believe that the revised version of the work is in a significantly better shape.

Comments on the paper

3 comments
AuthorsOpenReview ↗

A note on constructive scientific discussion

Dear Reviewer PFhP, Reviewer GCzi, Reviewer qKR7, and Area Chair,

We value ICLR's open discussion and appreciate the time and expertise that all reviewers and the AC contribute. We hope to move beyond this unexpected disagreement and keep the discussion focused on the paper's evidence, contributions, and remaining technical questions. We welcome further technical questions and remain committed to a constructive discussion with mutual respect.

Dear Authors,

I appreciate your efforts to keep the discussion focused on the technical aspects of the paper.

As AC, I will ensure that the paper is assessed fairly, taking into account its technical contributions and the discussion.

Thank you again for your engagement throughout the process.

AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

Thank you and the reviewers for your time and constructive feedback. Below, we summarize the main contributions, the new results, and the reviewers’ feedback.

Core contributions and reviewers’ positive assessments

  • First systematic cross-family analysis. We identify spontaneous query heterogeneity across 1D visual tokenizers and connect it to semantic information and localized reconstruction. All three reviewers recognized its value: PFhP acknowledged the evidence for role differentiation, GCzi called it “the most interesting part,” and qKR7 described it as “informative.”

  • Analysis-driven design with controlled validation. These findings led us to test token-dependent processing of heterogeneous queries. All three reviewers acknowledged empirical gains. Controlled comparisons, including the 12 configurations in Figure 5, show improvements in reconstruction, semantic representation, and generation at comparable active computation. The added equal-total Dense and fixed-random-routing comparisons further separate the effects of capacity and learned routing.

  • Novelty beyond directly applying MoE. PFhP and GCzi highlighted the routing ablations as a strength. The experiments show substantially larger representation gains for query-based than patch-based tokenization. They also reveal tokenizer-specific tradeoffs: the tested balancing strategies reduce joint quality, while decoder routing improves LP but worsens reconstruction and generation. These findings provide practical guidance for routing in visual tokenizers.

Reviewer PFhP (rating: 4, clear acceptance; confidence: 4)

  • Concerns: Unclear analysis settings, robustness of attention analyses, experimental comparability, and missing related work.
  • Our response: Clarified the paired-checkpoint settings in §3; added layer-wise and spectral analyses, threshold tests, and matched-input understanding evaluations; expanded related work and Appendix E’s discussion of LVLM visual interfaces.
  • Reviewer’s feedback: PFhP confirmed that a “significant majority” of the concerns had been addressed and stated: “I will be revising my score to a clear accept.”

Reviewer GCzi (rating: 3, weak acceptance; confidence: 1)

  • Concerns: Qualitative evidence, threshold robustness, total capacity, balancing strategies, higher resolution, and text/small-face reconstruction.
  • Our response: Added the requested controls, analyses, and visual comparisons. MoETok leads the six evaluated 1D tokenizers in both text and face reconstruction on TokBench. The paired comparison also shows that MoETok’s gains persist at 1024×1024: LP improves by 6.60 points and gFID decreases by 16.9%, with both models evaluated after 40,000 training iterations at matched forward computation. Figure D.3 now includes text and multi-person face comparisons.
  • Reviewer’s feedback: GCzi acknowledged the total-parameter-matched Dense and balancing experiments, called the 512×512 experiment a “useful controlled check,” and raised the score after the small-face comparisons.

Reviewer qKR7 (rating: 3, weak acceptance; confidence: 4)

  • Concerns: Novelty beyond standard MoE and the benefit of learned routing over fixed random routing.
  • Our response: Explained how the cross-family analysis motivates conditional processing. Revised Table 3 adds the requested fixed-random-routing comparison: learned routing improves LP by 3.74 points and lowers gFID from 11.24 to 9.59 at matched expert capacity and computation.
  • Reviewer’s feedback: We have not yet received a follow-up reply.

We hope this summary helps your assessment of the revised paper. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.