The work has the following primary weaknesses:
- W1: Problems with the Motivation and Narrative
- The biggest issue I can see with these parts is that these sections lack stand-alone precision. There are many eye-appealing visualizations all of which seem to support the narrative of the work, sure, but precisely which model, on which data and under which settings was used to generate those is not detailed in the main text (e.g. for Figure 2 and Table 1). Although I understand that the space limitations might have caused this, as a reader, it is very hard to fully understand the substance behind the visualizations in a clear manner. Accordingly, any revision of the work should clearly and concretely identify the core settings behind the visualizations more clearly. As it stands, Section 3 is very hard to follow.
- Another narrative issue to me stems from going from the motivation on the dichotomy of token roles vs the MoE structure. The statement "These findings motivate token-dependent parameter allocation: each token selects a subset of experts from a larger parameter pool, with active computation comparable to dense processing." reads a lot more like a self-fulfilling prophecy: There could be diverging roles within tokens, which is an "emergent" property as the authors term, but how does this necessitate directing the computation for each subset? Is there a fundamental gap caused by the lack of computation reserved for each sub-group of tokens? There are some linear probing/rFID numbers on Table 1 to demonstrate how MoETok improves over the Dense baseline, though the numbers could very well be confounded since (i) it is not precise what the difference between Dense vs MoeTok is and (ii) MoETok might have more number of total parameters (which from a latency perspective doesn't add much but has other computations overhead, e.g. memory) or has gone through a different training set-up. These points should ideally be clarified to tie the motivation to the proposed method.
- W2: Problems with the Attention Analyses and Section 3
- The work cites several works for the attention analyses it presents, which is appreciated. However, this doesn't justify certain design choices taken by the authors to motivate certain design choices that were made.
- Notably, the authors state that they average the Vq values across layers. However, averaging across depth in such a way could be problematic, given existing works in the literature note that certain layers in vision transformers are subject to having outlier representations, e.g. register/sink tokens emerging only after a certain index [A, B]. Ideally, to justify such aggregation, the authors should present how these patterns across different layers vary (in the form of a matrix of Vq values across tokens/layers) and whether there are any outlier layers. If the behavior across the layers isn't fully consistent, then the utilized aggregation operation could be biasing the observations made in the work.
- Finally, the work claims that content tokens capture "high-level semantic information", which is primarily demonstrated by the indirect attention patterns of such tokens. Although I acknowledge the common use of attention patterns for such purposes in the literature, I believe that the work's analyses could be strengthened by principled frequency analyses, e.g. by analyzing frequency-band or spectra captured by each token similar to the analysis conducted in existing works [C, D]. This is more of a nice-to-have and could demonstrate to what extent the content tokens are capturing the "high-level semantic information" or the low-frequency layout of the underlying image features.
- W3: Problems with the Experimental Analysis
- The most important issue with the experiments is that many of them aren't apples-to-apples: In the primary result table where MoETok is compared against the baselines, MoETok's only matching setting with them is its number of tokens. Although I acknowledge that MoETok has impressive gains over GigaTok-XL-XXL, it seems to bring very modest improvements over the primary baseline of GigaTok on major image generation measures, e.g. +0.06 rFID and +0.38. This is despite MoETok having substantially more trainable parameters compared to GigaTok (845M vs 505M), while only matching it at the active number of parameters.
- This issue is also repeated across several different result tables, e.g. Tables 7, 8 and 9, where the rows on the table follow different backbones, different training data, different hyper parameter settings and more. Although I understand that it is simply infeasible to control for each of these settings, the tone of the narrative as it stands is over-emphasizing the gains of MoETok, whereas it should have been modestly stating that those tables are merely there as a proof-of-concept. Any other claim otherwise should ideally be substantiated with controlled and matched experiments.
- Finally, the results under Table 6 on multimodal understanding have more of a heuristic value on demonstrating that MoETok does not entirely erase the understanding capabilities, rather than preserving them effectively. Many of the benchmarks presented in Table 6 with strong retention are known benchmarks which with very small number of visual tokens, it is possible to retain most or all of the performance, as shown in visual token compression works from the multimodal understanding literature [C, D, E]. For the benchmarks on Table 6 that aren't primarily reported in these works, for MMB and MMMU, MoETok has a very heavy negative delta against Qwen3.5-4B, −40 pts at MMMU and −14 pts at MMB, which further reduces the significance of results reported under Table 6.
- W4: Missing Related Works from the Literature
- As a final weakness, the work is overlooking many existing works from the literature on query-based (or 1D) visual tokenization for multimodal applications. Although I acknowledge the application domain difference between these works and MoETok, for good scientific practice, acknowledging these work is a nice-to-have.
- These could be listed as Matryoshka Query Transformers [E] which is the predecessor of FlexTok with a nearly-identical mechanism although used for understanding and other related works from the vision-language understanding literature [D, F] which also follow from a role-based dichotomy of visual tokens akin to MoETok should at least be cited.
- Finally, an existing arXiv work that came out in March 2026 [G] seems to have a similar promise with MoeTok, although importantly for understanding and not image generation.
[A] Jiang, N., Dravid, A., Efros, A., & Gandelsman, Y. (2026). Vision Transformers Don't Need Trained Registers. Advances in neural information processing systems, 38, 56557-56595.
[B] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024, May). Vision transformers need registers. In International conference on learning representations (Vol. 2024, pp. 2632-2652).
[C] Pan, Q., Dou, Z., & Li, S. (2026). Frequency-modulated visual restoration for Matryoshka large multimodal models. In Conference on Computer Vision and Pattern Recognition 2026.
[D] Kuzucu, S., Tonioni, A., Lup, V., Schiele, B., Tombari, F., & Naeem, M. F. (2026). PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding. arXiv preprint arXiv:2605.30126.
[E] Hu, W., Dou, Z. Y., Li, L. H., Kamath, A., Peng, N., & Chang, K. W. (2024). Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37, 50168-50188.
[F] Zhang, J., Guo, X., Cai, K., Lv, Q., Fan, Y., Chai, W., Wang, J., & Wang, K. (2026). Hybrid token compression for vision-language models. In Conference on Computer Vision and Pattern Recognition 2026.
[G] Li, Z., Li, Y., Fang, F., Takezoe, R., Bo, Z. H., Qian, C., ... & Long, K. (2026). QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression. arXiv preprint arXiv:2603.21232.
Author Response: Research Contributions and the Value of Learned Routing
Thank you for recognizing our experiments across scales and token budgets and for suggesting the fixed-random-routing comparison. We appreciate the constructive discussion. Below, we clarify how the analysis motivates our design and present the additional evidence on learned routing.
We respectfully disagree that “mainly applies standard sparse MoE” captures the full contribution. We present the first systematic cross-family analysis of spontaneous query heterogeneity in 1D visual tokenizers.
Figure 1shows recurring differentiation across architectures and training methods; masking and subset probes directly connect these behaviors to semantic and localized reconstruction preferences (Figure 2/Table 1). This motivates a testable hypothesis: task-learned transformations can better process heterogeneous queries, with unchanged supervision. InTable 2, applying MoE directly to a patch-based 2D tokenizer yields substantially smaller representation gains than in query-based 1D tokenization (LP +2.43 vs. +11.50 points, each against its matched Dense baseline).Figure 5extends the quality–compute evidence across 12 configurations, and revisedTable 3separates capacity from learned expert selection. Full ImageNet and unified-model evaluations establish downstream value. The study also identifies tokenizer-specific choices: enforced balance worsens joint quality, while decoder routing trades higher LP for worse reconstruction/generation (originalTable B.1;Appendix C.1/Table C.1). The contribution thus connects a cross-family finding, analysis-led design and controlled validation, rather than relying on component novelty alone.Learning expert selection improves reconstruction, linear separability and generation at the same capacity and computation, with similar zero-shot accuracy. Following your suggestion, we randomly initialize and freeze the router while training the remaining network, matching expert count, width, Top-k and active/total capacity; assignments still depend on token inputs. Revised
§4.2/Table 3includes the pair reproduced below (199M/313M active/total encoder parameters; 155 encoder GFLOPs; 641 MiB peak FP16 forward memory at batch 1; shared 303M decoder; unguided gFID). Learning the router improves LP by 3.74 points and lowers rFID/gFID by about 0.144/1.65, showing that task-learned paths use the same available expert capacity more effectively.