acceptodds
Withdrawn from ICLR 2027

Learning to Route Patches: Mixture of Patch Embeddings Vision Tokenization for MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) traditionally rely on a fixed visual tokenization scheme, where all images are discretized using a single patch size, regardless of input complexity or task requirements. This design enforces a uniform trade-off between representation fidelity and computational cost, which we find to be suboptimal. In this work, we investigate the role of visual token granularity under a fixed token budget. Our results show that reducing patch size consistently yields larger performance gains than increasing image resolution at comparable computational cost, indicating that tokenization granularity is a more effective factor in improving multimodal reasoning. However, finer tokenization is not universally optimal, and different benchmarks exhibit distinct performance saturation points, suggesting that the best patch size varies depending on the input and task. Motivated by this, we propose Mixture of Patch Embeddings (MoP), an adaptive visual tokenization framework that jointly trains multiple patch embedding modules and learns a lightweight routing policy to select the most suitable patch size for each input. MoP offers instance-wise control over representation granularity, enabling flexible performance optimization. Extensive experiments across diverse multimodal benchmarks demonstrate that MoP consistently outperforms fixed patch size baselines, delivering the best performance on 7 benchmarks and reducing FLOPs by over 1.4× compared to fixed patch tokenization. Our results highlight the potential of adaptive tokenization in advancing MLLMs, where the flexibility of patch size selection enhances model performance without necessitating significant efficiency trade-offs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RFzqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a Mixture of Patch Embeddings (MoP) method for MLLMs. Given an input image, a classifier is trained to select from 4 patch sizes (8, 10, 12, 14) to employ in the visual tokenization scheme, as the per-image patch size router for MLLMs. This patch size classifier is trained after a multi-patch tokenizer adaptation scheme using 4.3M image-text and OCR captioning samples and 2.9M fine-grained images. The MoP method improves the baseline method using Qwen3-VL-4B on 7 out of 9 benchmarks. These results suggest that employing a small patch size in the tokenization for images with fine-grained visual details improves the performance of MLLMs.

Critical strengths

The paper performed a good empirical study on the patch sizes vs. image resolution in terms of the performance and efficiency of MLLMs, which is a descent motivation for the proposed the MoP method.

The proposed MoP is interesting and reasonable, which demonstrates competitive performance on a diverse set of multimodal benchmarks.

Critical weaknesses

The finding in the paper is not a surprise and quite reasonable, i.e., using different patch sizes in the tokenization for individual input images shall improve the visual understanding performance. The major concern is how significant of this finding and the technical contribution, if the improvement of MoP is general and significant enough to justify the additional implementation complexity.

In Fig.2(c), the differences of the performance using different patch size in the tokenization are not that significant, e.g., 98.41% (patch size 8) vs. 98.76% (patch size 10). The performance gain of MoP over the baseline Qwen3-VL-4B backbone in Table 1 is reasonable though not that significant neither.

The training set of the patch size classifier is just 20K images, after using (4.3M+2.9M) images in the multi-patch tokenizer adaptation, which is surprisingly small. So the performance tends to be saturated even using larger training set for this router?

Overall, the paper identified a niche problem of patch size in visual tokenization and proposed a reasonable divide-and-conquer scheme using an additional per-image classifier/router, which could be applied to other hyper-parameters in MLLM implementations.

Questions for authors

See the comments on the weakness.

Reviewer_dN98OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the effect of visual patch size on the accuracy and computational cost of multimodal large language models. Its pilot experiments suggest that reducing patch size can be more effective than increasing image resolution at comparable token budgets, while the best patch size varies across benchmarks and examples. Motivated by these observations, the authors propose Mixture of Patch Embeddings (MoP), which adapts multiple patch embedding branches within a shared backbone and trains an image-conditioned router to select one branch per image. Routing targets are obtained by evaluating the answer-generation loss under each candidate patch size.

The reported results show that MoP can retain accuracy comparable to the finest fixed patch size while using approximately 1.4× fewer FLOPs. The appendix also reports latency improvements and evaluates the framework on LLaVA-OneVision-7B in addition to the primary Qwen3-VL-4B backbone.

Critical strengths
  1. The problem is practically relevant. Allocating visual computation according to the input is a useful alternative to uniformly applying the finest tokenization. The investigation of patch size versus image resolution provides a concrete motivation for the method.

  2. The framework is straightforward and offers a useful empirical tradeoff. Only one patch embedding branch is activated during inference. The reported efficiency benefit is supported by per-benchmark FLOPs and latency measurements, rather than FLOPs alone.

  3. The paper includes several informative controls. These include continued training of the original architecture on matched data, fixed-branch evaluation of a shared multi-patch backbone, comparison with the AdaPatch routing strategy on the same backbone, and transfer to a second MLLM architecture. These experiments provide meaningful initial evidence for the approach.

Critical weaknesses

1. The advantage of learned routing over simpler alternatives is not established at comparable computational cost.

The methodological contribution builds on established ideas: FlexiViT supports multiple patch sizes within one backbone, AdaPatch adapts patch size according to image properties, and APT explores different patch sizes within an image. MoP's distinguishing component is its use of backbone-specific answer losses to supervise an image-level router, together with its adaptation and empirical analysis in MLLMs. This is a sensible design, but its incremental value needs a more controlled assessment.

Table 8 is useful because it compares MoP and the AdaPatch routing strategy on the same adapted backbone. However, it reports accuracy without the corresponding token counts or computational costs. The average accuracy advantage of 82.6 versus 79.9 therefore does not establish a better accuracy–cost tradeoff: the methods may allocate different amounts of computation.

A comparison at matched average FLOPs, or a small set of accuracy–FLOPs operating points, would address this concern. A complementary baseline would preserve the learned router's patch-selection proportions while randomly reassigning those choices across examples. Because image dimensions can vary, this comparison should also report and control actual computational cost rather than assuming that equal routing proportions imply equal FLOPs.

2. The available ablations do not fully explain the gains of the main model.

Figure 5 shows that most of the accuracy improvement over the original backbone is already present after multi-patch adaptation. Relative to the best fixed branch, the router adds 0.5, 0.1, and 0.1 percentage points on RealWorldQA, AI2D, and MMStar, respectively, and loses 0.4 points on ChartQA. These small accuracy differences do not negate the router's value, since maintaining accuracy at lower cost is itself a useful result. They do, however, make it important to separate the benefits of backbone adaptation from those of routing.

The relevant experiments use different training recipes. The main model uses 4.3M pretraining and 2.9M instruction-tuning samples; Figure 5 uses 4.558M pretraining and 3M OCR-oriented samples; and Table 2 uses 4.558M pretraining and 858K SFT samples. These differences are disclosed in the appendix, and the individual comparisons can still be informative. However, their results cannot be combined directly into a complete attribution of the main model's gains. Table 1 also mixes official and self-evaluated baseline scores, which the authors appropriately distinguish from controlled comparisons.

The most useful addition would be one consolidated comparison under the main model's training recipe and evaluation protocol: continued training of the original architecture, fixed branches of the adapted backbone, and heuristic and learned routing, with both accuracy and cost reported. The existing two-backbone evaluation is a strength; resolving this attribution issue is more important than adding a broad range of architectures or model sizes.

3. The strength and consistency of the instance-level routing evidence need clarification.

Table 3 provides positive evidence for instance-level selection: the router exceeds uniform random selection and the best fixed patch size on all three confusing subsets. However, its margins over the best fixed choice correspond to approximately four additional correct answers on AI2D, one on RealWorldQA, and two on MMStar. These are small absolute differences. Paired uncertainty estimates or tests based on per-example correctness would help establish their robustness. Deterministic decoding removes sampling variability at inference, but does not resolve uncertainty due to the finite evaluation set or router training.

There is also a specific correspondence issue between Figure 5 and Table 3. Under the setup described in Appendix A.1.4, routing cannot change correctness on examples where all fixed branches agree. If both results use the same checkpoint, evaluation set, and protocol, the net gain in correct answers over a fixed branch must therefore be identical on the confusing subset and the full dataset. For RealWorldQA, Table 3 implies approximately one additional correct answer over p=8. On the 765 examples indicated in Figure 3, this would yield approximately 0.13 percentage points of improvement. Figure 5 instead reports 73.1 to 73.6, a gain of 0.5 points.

This may reflect different experimental settings rather than an error, but the paper should identify which checkpoints and protocols produced these results and reconcile the numbers if they are intended to describe the same setting.

4. The task-dependent interpretation goes beyond what image-only routing demonstrates.

Equations (3)–(6) define a router conditioned only on the image, whereas its supervision depends on the image, question, and target answer. This is a valid learning setup: an image-only router can learn patch preferences averaged over the questions associated with similar images. However, it cannot change its decision when the question changes for the same image.

Consequently, dataset-dependent routing distributions do not establish adaptation to the requirements of the current question. Appendix A.3.2 further states that the router training set retains only the shortest question per image and filters questions longer than 40 tokens, so this training setup does not directly examine competing granularity requirements across questions about the same image.

The paper should distinguish image-dependent adaptation from question-dependent adaptation. A same-image, multiple-question analysis would test whether this distinction matters in practice. A question-conditioned router would be an informative extension, but is not necessary if the claims are appropriately scoped.

Questions for authors
  1. Reliability of the routing target. How well does the ordering of patch sizes by teacher-forcing loss agree with their ordering by actual answer correctness? This would directly validate the proxy used to supervise routing, especially when candidate losses are close.

  2. Efficiency measurement. Table 11 reports latency speedups of 1.09×–1.22×, alongside larger FLOPs reductions. Please specify the inference hardware, batch size, timing scope, and whether the measurements include router execution and image preprocessing. The training batch sizes in Table 6 do not resolve these questions.

  3. Budget control. The routing objective contains no explicit cost penalty or budget constraint. This does not invalidate the reported efficiency gains, but it limits claims about controllable allocation under different budgets. Cost-aware labels or a preference for cheaper branches when their losses are nearly tied would be useful future comparisons.

  4. Scope of the pilot findings. Appendix A.1.1 places the patch-size-versus-resolution study on LLaVA-UHD-v3 with a particular training recipe. Please make the image resizing settings and resolution/patch-size pairs explicit, and scope the conclusions to the evaluated conditions. The current evidence does not establish that refining patches is generally preferable across preprocessing and pretraining regimes.

The most useful additions would be a controlled accuracy–cost comparison under the main training recipe, uncertainty analysis for routing gains, and clarification of the relationship between Figure 5 and Table 3. A narrower interpretation of task-dependent routing would also make the claims better aligned with the demonstrated capabilities.

Reviewer_CrxyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mixture of Patch Embeddings (MoP), which adaptively adjusts the patch size of the vision encoder in MLLMs. MoP selects a suitable patch size from {8, 10, 12, 14} for each input sample. Evaluations across nine benchmarks demonstrate consistent improvements in VQA performance over the base MLLM, along with considerable generalizability and efficiency benefits compared with fixed patch size strategies.

Critical strengths
  • The motivation is clear and straightforward. The observation that reducing patch size is generally more effective than increasing image resolution under comparable token budgets, while smaller patches do not always improve performance and optimal granularity varies across tasks and samples, naturally motivates MoP.

  • The method is concise and easy to follow. The router design and training strategy are intuitive and reasonable. The experiments distinguish the contributions of MoP from those of fixed patch size choices and additional training data.

Critical weaknesses
  • The core idea of adaptively adjusting the vision encoder’s patch size is closely related to AdaPatch. Although a comparison with AdaPatch is provided in Appendix Table 8, the detailed comparison is absent from the main text. Given this similarity, AdaPatch should be included in Table 1. Furthermore, the proposed method appears to have some disadvantages relative to AdaPatch. First, its patch size choices are restricted to {8, 10, 12, 14}, whereas AdaPatch supports integer patch sizes within [6, 56]. Expanding MoP’s candidate set would significantly increase the training computation required for the exhaustive patch rollouts used to supervise the router. Second, AdaPatch uses a simpler information density criterion to select the patch size, whereas MoP relies on DINOv3 features. AdaPatch therefore appears more computationally lightweight and intuitive. The authors should clarify the differences between MoP and closely related adaptive patch size methods to better establish its novelty and contribution.

  • Although the quantitative results demonstrate MoP’s advantages in selecting suitable patch sizes across benchmarks, the paper lacks a deeper analysis of how routing decisions depend on the image features extracted by DINOv3. The case studies in Appendix A.8 suggest that questions involving finer visual details benefit from smaller patches. Additional case studies could help substantiate this observation. More critically, the paper does not explicitly explain why different benchmarks exhibit different patch size preferences, which is central to motivating MoP. An analysis of the data distributions across benchmarks, particularly differences in visual information richness reflected in DINOv3 features, could provide further insight and strengthen the grounding of this work.

  • Both the abstract and conclusion emphasize MoP’s FLOPs advantage over fixed patch size strategies. However, the supporting evidence is primarily presented in the appendix. Including this evidence in the main text would make the paper more self-contained.

Questions for authors
  • What image resolution is used in the experiments?
  • How robust is MoP across different parameter scales of Qwen3-VL?
  • A deeper analysis of the visual evidence underlying MoP’s patch size decisions, together with a clearer comparison against related patch size adaptation methods, would strengthen the significance of this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.