acceptodds
Withdrawn from ICLR 2027

MatryoshkaFusion: Multi-Granularity Visual Feature Fusion within Patch Regions for Multimodal Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-granularity understanding is essential for visual comprehension, as the human visual system processes visual signals at multiple granularities. However, multi-granularity understanding mechanisms remain scarce in MLLMs, a mainstream paradigm in vision-language understanding. A limited number of studies have introduced multi-granularity understanding to accelerate inference at the cost of some performance degradation, leaving the potential for enhancing visual understanding largely unexplored. In this paper, we propose MatryoshkaFusion, a novel Matryoshka multi-granularity method that fuses multi-granularity features within patch regions, thus enhancing the model's visual understanding ability without increasing the number of LLM input tokens or the volume of training data. Furthermore, we visually demonstrate and analyze the weight distribution of different granularities, offering a new interpretable perspective on the mechanism underlying multi-granularity visual features’ enhancement of MLLMs’ visual understanding ability. To validate the effectiveness of our method, two sets of experiments are conducted: (1) training a specialized model on the ScienceQA dataset to rapidly assess the feasibility of our method; (2) training a multimodal chatbot and evaluating its zero-shot performance across multiple benchmarks. In both experiments, our model achieves better results than existing advanced multi-granularity methods used in MQT-LLAVA and LLaVA-M.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal large language models, Matryoshka representative learning, Multi-granularity understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TKPLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MatryoshkaFusion, which takes patch features extracted from the vision encoder of an MLLM and reconstructs them into a 2D spatial layout and applies convolutions with different kernel sizes to generate fine-to-coarse visual features. Each coarse feature is mapped back to its original patch location, and the model uses the text instruction to determine which level of granularity is most relevant at each position. These features are then fused into a single visual feature per position. Importantly, the total number of visual tokens passed to the LLM remains the same as in the baseline. The proposed method outperforms both the baseline and existing multi-granularity approaches on ScienceQA. It also shows consistent improvements across most multimodal benchmarks and zero-shot settings.

Critical strengths
  • Unlike previous methods such as MQT-LLaVA and LLaVA-1.5-M3, which mainly focus on improving efficiency by reducing the number of visual tokens, this paper aims to improve visual understanding itself by leveraging multi-granularity information, which is well-motivated problem setting, and it clearly distinguishes the proposed approach from methods that simply focus on visual token reduction.

  • The performance improvements are consistent. MatryoshkaFusion shows substantial gains across most benchmarks, enhancing visual representations without increasing the number of visual tokens fed into the LLM is a practical advantage of the proposed method.

Critical weaknesses
  • It is unclear whether the performance improvements come from multi-granularity information or simply from the additional convolution layers. The gains may be attributed to the extra convolutional layers rather than the proposed multi-granularity design. This distinction needs to be further investigated and discussed.
  • The paper justifies the ScienceQA experiments as a means of rapidly validating the proposed method, but this rationale is not sufficiently convincing, particularly because the ScienceQA and Multimodal Chatbot experiments use different model configurations. Since the main ablation studies are conducted only on ScienceQA, it remains unclear whether their findings generalize to the primary Multimodal Chatbot setting. Furthermore, Table 4 reports results across different backbones without their corresponding baselines, making it difficult to assess the consistency of the improvements.
  • The paper claims to improve visual understanding through multi-granularity feature fusion, but does not report results on benchmarks such as InfoVQA, and SP-DocVQA, which involve small text and complex document layouts. Since coarse-grained features may lose fine-grained textual details, it remains unclear whether the proposed adaptive fusion is effective for information-dense images.
  • The paper does not evaluate whether the proposed method preserves the original LLM's text-only capabilities after multimodal fine-tuning. Since the model is trained on multimodal instruction data, it remains unclear whether the improvements in visual understanding come at the cost of catastrophic forgetting in language understanding and reasoning tasks.
Questions for authors
  • How can the authors demonstrate that the performance improvements originate from multi-granularity feature fusion rather than the additional convolutional layers? How does varying the number of granularity levels affect the model's behavior? Visualizations showing how the granularity heatmaps change with different numbers of levels, or qualitative examples illustrating how these changes improve the model's generated answers, would further strengthen the proposed method.
  • Could the authors clarify the rationale for using different LLM backbones and training configurations for ScienceQA and Multimodal Chatbot? Additional experiments using Mistral-v0.2-7B for Multimodal Chatbot and Qwen2.5-7B for ScienceQA, with the corresponding baselines in Table 4, would help demonstrate consistent improvements across different model configurations.
  • How well does the proposed method generalize to text-dense images containing small text and complex document layouts? Could the authors provide additional evaluations on benchmarks such as InfoVQA and SP-DocVQA to verify whether adaptive multi-granularity fusion preserves fine-grained textual information.
  • Could the authors evaluate the model on text-only benchmarks before and after multimodal fine-tuning to determine whether the proposed method causes catastrophic forgetting of the original LLM's language capabilities?
Reviewer_YafYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper fuses coarse-to-fine visual features back into each patch, keeping the token count and the training data fixed, and asks whether that improves an MLLM. The contrast is with MQT-LLaVA and LLaVA-M³, which use multi-granularity to save tokens and accept some accuracy drop. The question is reasonable, but the main results do not fully demonstrate the effectiveness of the proposed method. Please see reviews as follows.

Critical strengths

The problem is well posed. Multi-granularity features in MLLMs have mostly been used to drop tokens and speed up inference. Asking whether the same features can improve perception at a fixed token budget is a real shift.

The module sits between the encoder and the projector, and the fused sequence stays length n^2. If the gain is real, this is a cheap plug-in.

Table 6 is the one comparison with a matched backbone. On 10% of the instruction data, the method beats the baseline, MQT, and M³ on VQAv2, SEED, GQA, and VizWiz. The heatmaps in Figure 4 also match the design: question-relevant regions receive finer granularity.

Critical weaknesses

The headline tables do not control the setup. Table 7 compares a 224-resolution Qwen2.5-7B adapter model with published 336-resolution Vicuna-7B full fine-tunes, while calling the pipeline the same. The jumps on CMMMU (21.8 to 39.6) and MMBench-CN (58.3 to 68.8) look like a language-model swap; VQAv2 is a tie at 78.5. In Table 5 the matched baseline is already 91.22, so 94.35 after 20 epochs on ScienceQA cannot be credited to the module. The missing row is full 665K data, the same Qwen2.5-7B, the same resolution, with the module on and off.

The name does not match the mechanism, and the method is not specified tightly enough to reimplement. MRL and M³ make the coarse representation usable on its own. Here every scale is mixed by a convex combination, so nothing can be dropped at inference. Equation 3 never states the tensor shape of 𝐾 or the softmax axis; Figure 2 draws Key and Query nets, while the text says there are no extra parameters. Whether the kernels are learned convolutions or fixed pooling is also unspecified. Mini-Gemini already fuses multi-scale features inside a fixed token budget and is absent from the related work.

The ablations do not support the story in the text. Removing the weight network costs 0.75 points, so the gain comes from having multi-scale features, not from instruction-adaptive weights. The baseline is 91.22 in Table 1 and 89.89 when all three modules are removed in Table 2. There is no leave-one-scale-out and no parameter-matched single-scale control. Table 3 measures cost against feature concatenation, not against the 256-token baseline. The only clean gain is on 10% of the data.

Questions for authors

See Weaknesses

Reviewer_SKamOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MatryoshkaFusion introduces instruction-conditioned, patch-level fusion of multi-granularity visual features for multimodal large language models. Starting from a single visual encoder feature grid, the method constructs coarser representations using strided convolutions, spatially replicates them to the original grid resolution, and combines them through per-patch weights conditioned on visual features and averaged instruction embeddings. The fused representation preserves the original number of visual tokens and does not require additional image–text training pairs.

The paper evaluates the method through specialized training on ScienceQA and multimodal instruction tuning followed by evaluation across several benchmarks. It reports 94.35% accuracy on the ScienceQA image subset and improvements over a matched baseline and two existing multi-granularity approaches in a reduced-data chatbot setting. Additional validation includes component ablations, alternative encoder and language-model configurations, efficiency measurements against feature concatenation, and visualizations of granularity weights.

Critical strengths
  1. Spatially aligned fusion preserves the visual-token budget while incorporating contextual information at multiple granularities.

    • The method retains the original fine-grained visual features and combines them with region-level representations rather than replacing them with compressed tokens.
    • This is a practical design choice: it allows representation enrichment without extending the language model’s visual input sequence or multiplying the number of training pairs.
    • The fusion-versus-concatenation experiment further supports the value of combining these features before language-model processing.
  2. The reduced-data, matched-backbone experiments provide encouraging evidence of effectiveness.

    • Tables 6 and 8 compare the methods using the same baseline architecture and instruction-tuning data fraction.
    • The proposed method improves performance across several visual question-answering and reasoning benchmarks, while matching the strongest result on AI2D.
    • These comparisons are more informative than cross-model leaderboard comparisons because they better isolate the effect of the added visual processing.
    • The component ablations also indicate that multi-granularity construction and adaptive weighting both contribute to the reported performance.
Critical weaknesses
  1. The experimental controls do not adequately establish the source or generality of the improvements.

    • The strongest full-data results in Table 7 compare models that differ in language-model backbone, resolution, and fine-tuning strategy. Consequently, these results cannot isolate the contribution of MatryoshkaFusion.
    • The more controlled chatbot comparisons use only 10% of the instruction-tuning data, leaving its incremental benefit under the complete training regime unresolved.
    • There is also an unexplained discrepancy between the ScienceQA baseline in Table 1, which achieves 91.22%, and the configuration with all proposed components removed in Table 2, which achieves 89.89%. This discrepancy materially affects the estimated improvement attributable to the method.
    • Moreover, Table 4 reports only augmented-model results, so it does not demonstrate improvements across alternative backbones.
    • Requested improvements:
      • Provide parameter-matched single-scale controls and simpler multi-scale fusion baselines to distinguish the proposed mechanism from additional capacity or generic contextual aggregation.
      • Reconcile the inconsistent ScienceQA baseline results.
      • Evaluate the method under the full instruction-tuning regime and across alternative backbones where feasible.
  2. The method is insufficiently specified, and the mechanistic conclusions exceed the evidence.

    • Equation 3 directly combines visual features with averaged instruction embeddings without explaining how their dimensions and representation spaces are aligned.
    • Figure 2 depicts query and key networks, but their architecture and parameterization are not described.
    • The convolution type—dense, grouped, depthwise, or another implementation—is unspecified, despite its potentially substantial effect on parameter count and computational cost.
    • The manuscript also states that only the projection network is updated during pretraining without clarifying when the feature-construction and weight-generation modules are trained.
    • These omissions obstruct reproducibility and make the parameter and efficiency claims difficult to assess.
    • Table 3 measures efficiency relative to concatenating multi-scale features, not relative to the original single-scale baseline, so it does not establish negligible end-to-end overhead.
    • Finally, positive softmax weights do not demonstrate that every granularity is indispensable, and qualitative heatmaps do not establish reasoning ability or causal relevance.
    • Requested improvements:
      • Specify the feature-alignment mechanism, query/key architecture, convolution implementation, and training schedule for all modules.
      • Report end-to-end overhead relative to the original single-scale baseline.
      • Conduct scale-removal interventions and instruction-conditioning controls to support claims about the necessity of each granularity and the causal role of instruction conditioning.
Questions for authors
  1. ScienceQA baseline discrepancy: Why does the ScienceQA baseline achieve 91.22% in Table 1 but 89.89% when all proposed components are removed in Table 2? Please identify any differences in architecture, initialization, training, or evaluation, and report variation across random seeds.

  2. Implementation details and parameter accounting: Please specify the convolution implementations, query/key transformations, feature dimensions, softmax axis, and which modules are trained at each stage. How do these choices reconcile with the reported trainable-parameter counts?

  3. Assessment-changing experimental request — controlled fusion comparisons: Can you provide a matched single-scale baseline under the full-data chatbot training regime, together with parameter-matched single-scale and fixed-weight multi-scale controls? These comparisons would help determine whether the main improvements are attributable to the proposed fusion mechanism rather than additional capacity or generic multi-scale aggregation.

  4. Assessment-changing experimental request — efficiency and instruction conditioning: How do performance and end-to-end runtime change relative to the original baseline, and what happens when instruction conditioning is removed or shuffled? These results would clarify both the practical overhead and the necessity of instruction-dependent weighting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.