- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MatryoshkaFusion, which takes patch features extracted from the vision encoder of an MLLM and reconstructs them into a 2D spatial layout and applies convolutions with different kernel sizes to generate fine-to-coarse visual features. Each coarse feature is mapped back to its original patch location, and the model uses the text instruction to determine which level of granularity is most relevant at each position. These features are then fused into a single visual feature per position. Importantly, the total number of visual tokens passed to the LLM remains the same as in the baseline. The proposed method outperforms both the baseline and existing multi-granularity approaches on ScienceQA. It also shows consistent improvements across most multimodal benchmarks and zero-shot settings.
Critical strengths
-
Unlike previous methods such as MQT-LLaVA and LLaVA-1.5-M3, which mainly focus on improving efficiency by reducing the number of visual tokens, this paper aims to improve visual understanding itself by leveraging multi-granularity information, which is well-motivated problem setting, and it clearly distinguishes the proposed approach from methods that simply focus on visual token reduction.
-
The performance improvements are consistent. MatryoshkaFusion shows substantial gains across most benchmarks, enhancing visual representations without increasing the number of visual tokens fed into the LLM is a practical advantage of the proposed method.
Critical weaknesses
- It is unclear whether the performance improvements come from multi-granularity information or simply from the additional convolution layers. The gains may be attributed to the extra convolutional layers rather than the proposed multi-granularity design. This distinction needs to be further investigated and discussed.
- The paper justifies the ScienceQA experiments as a means of rapidly validating the proposed method, but this rationale is not sufficiently convincing, particularly because the ScienceQA and Multimodal Chatbot experiments use different model configurations. Since the main ablation studies are conducted only on ScienceQA, it remains unclear whether their findings generalize to the primary Multimodal Chatbot setting. Furthermore, Table 4 reports results across different backbones without their corresponding baselines, making it difficult to assess the consistency of the improvements.
- The paper claims to improve visual understanding through multi-granularity feature fusion, but does not report results on benchmarks such as InfoVQA, and SP-DocVQA, which involve small text and complex document layouts. Since coarse-grained features may lose fine-grained textual details, it remains unclear whether the proposed adaptive fusion is effective for information-dense images.
- The paper does not evaluate whether the proposed method preserves the original LLM's text-only capabilities after multimodal fine-tuning. Since the model is trained on multimodal instruction data, it remains unclear whether the improvements in visual understanding come at the cost of catastrophic forgetting in language understanding and reasoning tasks.
Questions for authors
- How can the authors demonstrate that the performance improvements originate from multi-granularity feature fusion rather than the additional convolutional layers? How does varying the number of granularity levels affect the model's behavior? Visualizations showing how the granularity heatmaps change with different numbers of levels, or qualitative examples illustrating how these changes improve the model's generated answers, would further strengthen the proposed method.
- Could the authors clarify the rationale for using different LLM backbones and training configurations for ScienceQA and Multimodal Chatbot? Additional experiments using Mistral-v0.2-7B for Multimodal Chatbot and Qwen2.5-7B for ScienceQA, with the corresponding baselines in Table 4, would help demonstrate consistent improvements across different model configurations.
- How well does the proposed method generalize to text-dense images containing small text and complex document layouts? Could the authors provide additional evaluations on benchmarks such as InfoVQA and SP-DocVQA to verify whether adaptive multi-granularity fusion preserves fine-grained textual information.
- Could the authors evaluate the model on text-only benchmarks before and after multimodal fine-tuning to determine whether the proposed method causes catastrophic forgetting of the original LLM's language capabilities?