acceptodds
Withdrawn from ICLR 2027

Meta-MemGran: Rethinking Pixel-Level Visual Encoder via Meta-Memory for MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in multimodal large language models (MLLMs) has largely depended on **CLIP**-based visual encoders. While these models excel at capturing global semantic alignment, they often struggle to preserve fine-grained visual details. By comparison, the recent **DINOv3** has exhibited emergent capabilities in fine-grained pixel-level perception, but still lacks the higher-level semantic abstraction required for alignment with LLMs and multi-granularity reasoning. These complementary limitations reveal a fundamental bottleneck: existing visual representations struggle to simultaneously preserve perceptual details and organize them into high-level semantics. We argue that the key challenge lies not merely in the visual encoder, but in the lack of an adaptive mechanism for transforming dense visual evidence into representations at different levels of abstraction. This raises a central question: ***How can fine-grained perception be effectively bridged with high-level semantics for multi-granularity multimodal reasoning?*** To address this challenge, we propose **Meta-MemGran**, a **DINOv3**-based MLLM with meta-memory-driven representation learning. By structuring and dynamically aggregating visual information through meta-memory, **Meta-MemGran** bridges pixel-level details, visual structures, and semantic abstractions without explicit granularity selection. Extensive experiments show that **Meta-MemGran** achieves an accuracy improvement of ** 40%↑** and a hallucination reduction of ** 30%↓**. Code is available at [Meta-MemGran](https://anonymous.4open.science/r/Meta-MemBank-BA1F/).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Multi-Granularity Visual Understanding, Meta-Memory Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KfRGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Meta-MemGran, a framework that uses structured memory to improve the visual representations of DINOv3 for multimodal reasoning. The method builds four memory banks for global information, entities, spatial layouts, and relationships, and combines their features through prototype retrieval and memory aggregation. Experiments on several benchmarks show improvements in reasoning performance and hallucination reduction. The overall motivation is reasonable, and the results are promising. However, I still have some concerns about the novelty of the method, several implementation details, and whether the experiments sufficiently demonstrate the claimed adaptive multi-granularity capabilities.

Critical strengths
  1. The paper addresses a relevant challenge in MLLMs: bridging fine-grained pixel-level perception and high-level semantic reasoning.
  2. The proposed Meta-MemBank incorporates four components covering global context, entities, spatial layouts, and inter-entity relationships. This provides a structured approach to organizing visual information, and the combination of prototype retrieval and memory composition offers a potentially useful alternative to explicit granularity control through an external language model.
  3. The paper reports experiments across multiple benchmarks and language model backbones, including Qwen and Llama models of different sizes. The results show improvements over several visual encoder baselines, particularly on reasoning and GPT-based evaluation metrics.
Critical weaknesses

The authors highlight that Meta-MemGran can adaptively organize visual information without relying on an external LLM for granularity control. However, the method mainly builds on existing techniques, including online clustering, prototype retrieval, spatial relation modeling, and attention-based feature fusion. Similar ideas have been explored in prior work on visual token aggregation and memory-augmented learning. As a result, I find the novelty somewhat limited, as the main contribution seems to be the combination of four types of visual memory rather than a new mechanism for adaptive granularity learning.

The Entity Bank uses entity names extracted from paired captions to organize visual prototypes. However, these captions are not available during inference, where the model only receives an image and a question. I am therefore unclear about how the Entity Bank works at inference time. How are the relevant entity names identified, and how are the entity indices used for retrieval?

I am not sure whether the four memory banks capture sufficiently different information. In Eq. (4), the Global Bank averages all patch features, while the Entity Bank averages the cluster centers. These representations may be quite similar, making it unclear how the Entity Bank preserves individual object information. Similarly, the Relation Bank relies on linear combinations of cluster features and predefined spatial rules, which may capture basic spatial relationships but struggle with more complex interactions. Although the ablation results show that the four banks contribute to performance, further analysis would help clarify whether they actually capture distinct and complementary semantic information.

The paper claims that Meta-MemGran can adaptively organize visual information based on different task requirements, but I would like to see more evidence of this. For example, when the same image is paired with questions about the overall scene, individual objects, or spatial relationships, do the retrieved memory prototypes and the contributions of different memory banks change accordingly? It would be helpful to show how the retrieval results and aggregation weights vary across these questions to better support the claimed adaptive multi-granularity capability.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_maazOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes to structure visual representations for LLMs into a predefined set of "memories": The Meta-MemBank comprises "Global, Entity, Layout, and Relation Banks". The method is very involved and includes clustering visual reprsentations from a training database alongside spatial relations. The LLM not only gets the usual visual features but also those from these memory banks.

Critical strengths
  • the method is shown when starting from a pretrained MLLM
  • getting better visual reprsentations for LLMs is a core challenge and important to study both for academia and industry.
Critical weaknesses
  • The authors write and use as key to their motivation "Existing visual encoders exhibit complementary strengths" and mention that CLIP and SigLIP give good image-text alignment but struggle with other things. However, this mixes the impact of data and learning method. Works like WebDINO and [1] that it's mostly about the training data and not the objective.
  • the gains are very narrow on the established evaluation metrics like recall or BERTScore. The gains in GPT-as-judge are higher, but this is likely because of the training data that is used here is using the same LLM as the one used for judging (GPT4).
  • It's unclear why this set of "banks"/perspectives is chosen and how.
  • The method argues that these banks are complementary, but there is not evidence or methodological reason why this is the case.
  • Not compared against text-conditional memory models such as [2] which is also missing from the related works section
  • it's odd/confusing for readers, to call these memory banks when just dealing with a singular image.
  • "Key insights" are written in the methods section when there's no insights to be had. They are rather "TLDR" or "gists" but not insights.
  • the motivation is that the method doesn't require a LLM, but it seems that a) a parser is used and b) a text model is trained
  • confusing/wrong description: eqn 6 both terms are "local semantic support"
  • the method is quite complicated and involves additional processing and giving the LLM more tokens making comparisons not fair
  • Qwen2.5 and llama3.2 are very outdated models.
  • imprecise results: table 10 only describes results as "SOTA" without any evidence or numbers [1] Data or Language Supervision: What Makes CLIP Better than DINO? https://arxiv.org/abs/2510.11835
Questions for authors
  • if you start from a pretrained VLM, why do you need to retrain the LLM component via LoRA?
  • how does the method perform on better, more recent VLMs Qwen3, SmolLM, internvl, molmo ?
  • how do the LLM-judge scores differ with a LLM that isn't close to the one used in Flux-Reason, e.g. a big Qwen, or GEmini or Claude?
  • how much more FLOPs are being used with your vision encoding? how mny more tokens are given to LLM? if this is normalised, does the method still maintain a performance? e.g. if you instead give the additional parameters to just plain LoRA tuning, is it still better to maintain your memory banks in comparison?
Reviewer_pa4sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a DINOv3-based multimodal framework that aims to bridge fine-grained visual perception and high-level semantic reasoning. The method introduces a structured Meta-MemBank with four components: Global, Entity, Layout, and Relation, which organize visual features through clustering, prototype retrieval, and attention-based memory composition. The resulting representations are integrated into an MLLM using a two-stage training strategy involving projector alignment and LoRA adaptation. Experiments report improvements in downstream reasoning performance and hallucination scores compared with other visual encoder baselines, including Granulon. The authors also conduct component ablations, and qualitative analyses of object attributes, relationships, and spatial layouts.

Critical strengths
  1. The paper addresses the important challenge of connecting detailed visual representations with higher-level multimodal reasoning.
  2. Evaluations across different LLM backbones and on unseen datasets provide additional evidence of its potential utility.
  3. The separation into Global, Entity, Layout, and Relation memory banks provides an intuitive way to organize complementary visual information.
  4. The authors include both individual-bank and leave-one-out ablations. The complete four-bank configuration generally outperforms its reduced variants, providing evidence that the components contribute complementary information.
Critical weaknesses
  1. Limited novelty and insufficient differentiation from related work. The central motivation of bridging fine-grained visual perception and semantic understanding has been extensively explored in prior research, including Eyes Wide Shut (CVPR 2024) [1], X-Former (ECCV 2024)[2], and Reconstructive Visual Instruction Tuning (ICLR 2025)[3]. More directly, Granulon (CVPR 2026) already investigates DINOv3-based multi-granularity semantic abstraction using adaptive token aggregation and relation-aware grouping. While Meta-MemGran introduces structured prototype memory banks, many constituent operations, including clustering, similarity-based retrieval, and attention-based fusion, are established techniques. The paper does not sufficiently demonstrate the distinct technical advantages of its memory formulation over Granulon or simpler token aggregation alternatives.

  2. Although improving fine-grained visual perception is central to the paper's motivation, the evaluations primarily focus on general VQA, captioning, and GPT-based reasoning scores. The paper lacks targeted quantitative evaluations such as Object Counting (OC) and Multi-Class Identification (MCI) from What Makes for Good Visual Tokenizers for Large Language Models? [4], which were also used in X-Former [2]. These tasks are particularly relevant for validating the proposed Entity Bank and its ability to preserve object-level information. Similarly, compositionality evaluation using SugarCrepe-style forced-choice caption selection could directly assess object–attribute associations and spatial relationships. These evaluations are particularly relevant to the proposed Entity, Layout, and Relation banks. The qualitative examples in Appendix D are insufficient to establish systematic improvements.

  3. The paper does not isolate offline prototype memory from online clustering and attention-based aggregation. The framework also introduces additional visual representations and trainable fusion operations, which may contribute independently to performance improvements. Comparisons against online clustering alone, unstructured or random memory prototypes, and parameter- and token-matched aggregation baselines would better establish whether structured memory provides a meaningful advantage.

  4. The method describes the router as a non-parametric Top-K similarity-based retrieval operator, yet router parameters appear among the trainable components in the second training stage. This distinction requires clarification. Furthermore, the dual-alignment objective in Section 3.1 does not clearly specify the optimization procedure, and the mapping between composed memory representations and visual tokens is insufficiently explained.

  5. A central claim is that Meta-MemGran enables dynamic, task-relevant semantic aggregation without explicit granularity selection. However, the described retrieval procedure relies primarily on visual-feature similarity and geometry-based relational structures. The experiments do not adequately demonstrate whether different questions about the same image lead to different memory retrieval decisions or compositions. Without such analysis, it remains unclear whether the method genuinely performs question-adaptive semantic abstraction or augments visual representations with relatively static structured information.

  6. A controlled comparison of additional inference latency, memory consumption, and token usage against equivalent DINOv3 and Granulon configurations is missing. The authors should separately account for the costs of online clustering, memory retrieval, and composition before concluding that the framework provides favorable computational scalability.

References [1] Tong et al. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR, 2024.

[2] Sirnam et al. X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs. ECCV, 2024.

[3] Wang et al. Reconstructive Visual Instruction Tuning. ICLR, 2025.

[4] Wang et al. What Makes for Good Visual Tokenizers for Large Language Models? 2023.

Questions for authors
  1. How does the proposed memory mechanism differ fundamentally from Granulon's adaptive multi-granularity aggregation and prior approaches [1,2,3]? Clarify what specific benefit it provides ?
  2. Fine-grained perception evaluation: Can the authors provide quantitative results on Object Counting (OC) and Multi-Class Identification (MCI), as well as compositionality evaluations such as SugarCrepe using a forced-choice VQA protocol compatible with generative MLLMs? These would more directly validate the claimed improvements in object-level perception and relational understanding.
  3. Memory contribution: How does the method perform with online clustering and attention-based aggregation alone, without the offline memory prototypes? Can the authors provide parameter- and token-matched comparisons to isolate the benefits of the memory mechanism?
  4. Is the router entirely non-parametric, or are routing parameters optimized during training? How is the dual-alignment objective implemented, and how are memory representations integrated with the visual tokens?
  5. How are entity-conditioned prototypes selected at inference when reference captions are unavailable? Can the authors provide examples showing how retrieved memory prototypes and bank contributions change for different questions about the same image?
  6. Can the authors report individual-sample human/GPT agreement statistics and confidence intervals for the hallucination evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.