- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CoReMem, a framework for context-aware memory construction and query-aware retrieval in multimodal conversations. The method first constructs a memory bank of turn-level and segmented subimage documents by utilizing VLM prompts, object bounding boxes, SAM3 masks, and image-text descriptions. It then performs iterative retrieval by combining text and SigLIP2 similarities, running a VLM-based sufficiency check, and executing visual query expansion via DINOv3-7B feature matching. Experiments are conducted on the MemEye benchmark using Qwen models, reporting improvements over existing multimodal retrieval-augmented generation baselines.
Critical strengths
-
The paper presents a comprehensive, well-engineered, and end-to-end framework for multimodal conversational memory. By unifying dynamic text summarization, object detection, pixel-level SAM3 segmentation, multi-space indexing (using MiniLM, SigLIP2, and DINOv3-7B), and iterative query-aware visual retrieval, the system successfully bridges the gap between text-based dialogue tracking and local visual detail preservation. This provides a valuable architectural blueprint for building complex, agentic multimodal memory systems.
-
The empirical validation is supported by a dedicated benchmark (MemEye) featuring diverse multi-session tasks and extended context settings, demonstrating clear performance gains over existing multimodal RAG baselines. Furthermore, the paper exhibits commendable transparency in its documentation, providing extensive appendices that thoroughly detail system prompts, architectural notations, hyperparameter configurations, and qualitative case studies, which greatly benefits reproducibility and future research in this domain.
Critical weaknesses
-
Unfair comparison due to asymmetric encoder usage and expanded retrieval budgets. While all multimodal baselines are restricted to lightweight encoders like MiniLM and SigLIP2-base, CoReMem introduces a massive 7B-parameter vision foundation model (DINOv3 ViT-7B/16) for subimage and image feature indexing without providing an ablation to isolate its contribution. Furthermore, while baselines are strictly capped at 10 retrieved conversational turns, CoReMem dynamically expands its retrieved context up to 12-19 turns through iterative rounds, giving it an unfair context-length advantage in the main evaluation tables.
-
Incomplete cost and latency analysis that ignores heavy online compute overhead. Section 3.4 only reports the number of VLM completion tokens during memory construction, completely omitting the heavy input visual tokens, SAM3 segmentation costs, and DINOv3-7B inference overhead. More importantly, the paper lacks any evaluation of online query-time latency and token costs. Unlike single-shot vector retrievers, CoReMem requires serial multi-round VLM sufficiency checks, SoM visual marking, real-time SAM3 segmentation, and DINOv3 feature matching for every user query, making its practical deployment prohibitively expensive.
-
Internal design contradictions and weak component contributions. Although the system expends significant computational effort on extracting and describing segmented subimages during construction, Appendix C.3 explicitly notes that subimage descriptions are discarded during final generation and replaced by the original parent turn images. Additionally, ablation studies in Table 3 show that removing core components—such as the VLM sufficiency check, summaries, and iterative retrieval—often yields higher scores on complex multi-hop or fine-grained reasoning tasks, undermining the necessity of these modules.
-
Evaluation is restricted to a single small synthetic benchmark and lacks competitive re-ranking baselines. All experiments are evaluated exclusively on MemEye (371 total questions), with the extended tasks merely concatenating the exact same 371 questions across longer histories. Strong alternative strategies, such as simple two-stage re-rankers (MMRAG combined with a cross-encoder), achieve comparable performance to CoReMem in the full context setting (Figure 3) without requiring complex segmentation and multi-round VLM interactions, yet they are omitted from the main comparison tables.
Questions for authors
-
What are the exact online query-time latency and total token consumption (both input and output) of CoReMem compared to MMRAG per user query, and how do performance and latency change if CoReMem replaces DINOv3-7B with the same SigLIP2 encoder used by the baselines?
-
Given that subimage descriptions are ultimately discarded and final generation relies on original parent images (Appendix C.3), why is complex pixel-level SAM3 segmentation and multi-round mask verification necessary compared to simpler bounding-box crops?
-
Why do ablation results in Table 3 show that removing the VLM sufficiency check and iterative retrieval actually improves performance on several complex fine-grained tasks (X4, Y1, Y3), and how do you justify their inclusion as core contributions?
We sincerely thank the reviewers for their time and effort in providing thoughtful, detailed, and constructive feedback. We greatly appreciate their valuable insights and suggestions, which will help us further improve and strengthen our work. Although we have decided to withdraw this submission, we will carefully consider the feedback in our future revisions.