acceptodds
Withdrawn from ICLR 2027

CoReMem: Context-Aware Memory Construction and Query-Aware Retrieval for Multimodal Conversations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon multimodal conversational agents must recover relevant visual evidence from extended interaction histories. Existing memory systems typically convert images to text, potentially omitting visual details, or retrieve complete images using global embeddings, which can underrepresent small but relevant regions. Their retrieval procedures also commonly rely on a fixed top- budget, which may omit evidence needed for multi-hop reasoning. We present CoReMem, a multimodal memory framework for context-aware construction and query-aware retrieval. During construction, a vision-language model uses only the dialogue observed so far to select potentially useful regions, SAM3 produces the corresponding masks, and validated regions become independently indexed subimage documents linked to their parent turns. During retrieval, hybrid text–image ranking first retrieves an initial ranked set of documents drawn from at most dialogue turns. A VLM checks whether this initial context is sufficient for the user query. If not, CoReMem constructs visual queries conditioned on the query and images from the retrieved turns. It marks retrieved subimages as candidate query regions and uses DINOv3-based image-to-image retrieval to expand the context. The sufficiency check is repeated after each expansion to determine whether further evidence is needed. On MemEye with Qwen3.6-27B, CoReMem achieves LLM-as-a-Judge score of , outperforming the strongest evaluated baseline by . It also improves performance by – on settings with longer dialogue and more topic transitions, demonstrating the value of context-aware memory construction and query-conditioned visual expansion for long-horizon multimodal conversation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Conversational Memory, Agent Memory, Multimodal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fRzAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoReMem, a framework for context-aware memory construction and query-aware retrieval in multimodal conversations. The method first constructs a memory bank of turn-level and segmented subimage documents by utilizing VLM prompts, object bounding boxes, SAM3 masks, and image-text descriptions. It then performs iterative retrieval by combining text and SigLIP2 similarities, running a VLM-based sufficiency check, and executing visual query expansion via DINOv3-7B feature matching. Experiments are conducted on the MemEye benchmark using Qwen models, reporting improvements over existing multimodal retrieval-augmented generation baselines.

Critical strengths
  1. The paper presents a comprehensive, well-engineered, and end-to-end framework for multimodal conversational memory. By unifying dynamic text summarization, object detection, pixel-level SAM3 segmentation, multi-space indexing (using MiniLM, SigLIP2, and DINOv3-7B), and iterative query-aware visual retrieval, the system successfully bridges the gap between text-based dialogue tracking and local visual detail preservation. This provides a valuable architectural blueprint for building complex, agentic multimodal memory systems.

  2. The empirical validation is supported by a dedicated benchmark (MemEye) featuring diverse multi-session tasks and extended context settings, demonstrating clear performance gains over existing multimodal RAG baselines. Furthermore, the paper exhibits commendable transparency in its documentation, providing extensive appendices that thoroughly detail system prompts, architectural notations, hyperparameter configurations, and qualitative case studies, which greatly benefits reproducibility and future research in this domain.

Critical weaknesses
  1. Unfair comparison due to asymmetric encoder usage and expanded retrieval budgets. While all multimodal baselines are restricted to lightweight encoders like MiniLM and SigLIP2-base, CoReMem introduces a massive 7B-parameter vision foundation model (DINOv3 ViT-7B/16) for subimage and image feature indexing without providing an ablation to isolate its contribution. Furthermore, while baselines are strictly capped at 10 retrieved conversational turns, CoReMem dynamically expands its retrieved context up to 12-19 turns through iterative rounds, giving it an unfair context-length advantage in the main evaluation tables.

  2. Incomplete cost and latency analysis that ignores heavy online compute overhead. Section 3.4 only reports the number of VLM completion tokens during memory construction, completely omitting the heavy input visual tokens, SAM3 segmentation costs, and DINOv3-7B inference overhead. More importantly, the paper lacks any evaluation of online query-time latency and token costs. Unlike single-shot vector retrievers, CoReMem requires serial multi-round VLM sufficiency checks, SoM visual marking, real-time SAM3 segmentation, and DINOv3 feature matching for every user query, making its practical deployment prohibitively expensive.

  3. Internal design contradictions and weak component contributions. Although the system expends significant computational effort on extracting and describing segmented subimages during construction, Appendix C.3 explicitly notes that subimage descriptions are discarded during final generation and replaced by the original parent turn images. Additionally, ablation studies in Table 3 show that removing core components—such as the VLM sufficiency check, summaries, and iterative retrieval—often yields higher scores on complex multi-hop or fine-grained reasoning tasks, undermining the necessity of these modules.

  4. Evaluation is restricted to a single small synthetic benchmark and lacks competitive re-ranking baselines. All experiments are evaluated exclusively on MemEye (371 total questions), with the extended tasks merely concatenating the exact same 371 questions across longer histories. Strong alternative strategies, such as simple two-stage re-rankers (MMRAG combined with a cross-encoder), achieve comparable performance to CoReMem in the full context setting (Figure 3) without requiring complex segmentation and multi-round VLM interactions, yet they are omitted from the main comparison tables.

Questions for authors
  1. What are the exact online query-time latency and total token consumption (both input and output) of CoReMem compared to MMRAG per user query, and how do performance and latency change if CoReMem replaces DINOv3-7B with the same SigLIP2 encoder used by the baselines?

  2. Given that subimage descriptions are ultimately discarded and final generation relies on original parent images (Appendix C.3), why is complex pixel-level SAM3 segmentation and multi-round mask verification necessary compared to simpler bounding-box crops?

  3. Why do ablation results in Table 3 show that removing the VLM sufficiency check and iterative retrieval actually improves performance on several complex fine-grained tasks (X4, Y1, Y3), and how do you justify their inclusion as core contributions?

Reviewer_bfAiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets long-horizon multimodal conversational memory, where relevant visual evidence may be lost when images are represented only by captions or global embeddings. It proposes CoReMem, which constructs context-aware subimage memories and links them to their parent dialogue turns for fine-grained retrieval. During retrieval, CoReMem first performs hybrid text–image retrieval and then iteratively expands the context through VLM-based sufficiency checking and DINOv3-based image-to-image retrieval. Experiments on MemEye benchmark show that CoReMem outperforms other baselines with particularly strong gains on fine-grained visual evidence and multi-hop reasoning.

Critical strengths
  1. The paper addresses an important problem: preserve and retrieve fine-grained visual evidence in long-horizon multimodal conversations.
  2. The core motivation is clear, particularly the observation that important visual evidence may reside in fine-grained subimage regions that are not adequately preserved by global captions or whole-image embeddings.
  3. The empirical results are strong, with consistent gains over the strongest evaluated baselines on MemEye benchmark.
Critical weaknesses
  1. The proposed pipeline is relatively complex, while its practical cost is not sufficiently characterized. CoReMem relies on multiple VLM calls, SAM3-based segmentation and validation during memory construction, together with iterative sufficiency checking, and DINOv3-based retrieval at query time. However, the current cost analysis mainly reports completion-token usage, leaving the end-to-end latency, retrieval-time overhead, GPU memory footprint of the overall deployment unclear.
  2. The evaluation is limited to a single benchmark, which weakens the generality of the conclusions. Evaluation on additional multimodal memory benchmarks would make the conclusions more convincing.
  3. The baseline results lacks sufficiently in-depth analysis. In Table 1, relatively simple retrieval baselines such as TextRAG and MMRAG outperform most specialized memory systems by a considerable margin. Therefore, it remains unclear to what extent the observed gains of CoReMem can be attributed to the proposed method itself, but not benchmark-specific (MemEye-specific) effects or other implementation-level differences. A deeper analysis of this phenomenon would help better understand the actual contribution of CoReMem’s design.
Questions for authors
  1. Could the authors also provide an additive comparison starting from MMRAG and progressively introducing individual components, such as iterative retrieval and dynamic visual-query construction? Such an analysis would be helpful for understanding the contribution of each component to the overall performance.
  2. In current design, the subimage mainly serves as a fine-grained retrieval key, while the final response model still receives the original parent image and dialogue turn. Could the authors explain the motivation behind this design choice and whether directly using the retrieved subimage has been considered?
  3. For documents without images, why is simvissim_{vis} set to zero while text and visual similarities are still averaged with equal weights? This strategy effectively downweights the retrieval score of text-only documents, even when they have high textual similarity. Could the authors explain the motivation behind this design choice?

If the authors can adequately address these concerns, I am open to raise my score.

Reviewer_CvFTOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes COREMem, a framework for long-term multimodal conversational memory that combines context-aware memory construction with query-aware retrieval. During memory construction, the method leverages preceding dialogue context to identify salient image regions, segments them using SAM3, and indexes the resulting subimages as independent memory documents linked to their parent dialogue turns. During retrieval, the framework retrieves relevant memory documents using hybrid text -image similarity, assesses evidence sufficiency through a VLM, and performs iterative img2img retrieval using dynamically constructed visual queries when additional evidence is required. The authors evaluate COREMem on MemEye and extended-context variants using two Qwen3.6 models. The method achieves improvements over text-only and multimodal memory baselines, e.g., for fine-grained visual retrieval and multi-hop reasoning.

Critical strengths
  • I believe the paper addresses an important problem in long-term multimodal conversational memory. Although the individual components are not entirely novel, combining context-aware visual memory construction with query-aware iterative retrieval is a reasonable and practical approach to preserving and retrieving fine-grained visual information across long conversations.
  • The paper shows decent empirical improvements over existing baselines on MemEye with Qwen3.6-27B. I also appreciate that the authors provide extensive ablation studies to validate the contributions of individual components. Additionally, the human validation of the LLM-as-a-Judge evaluator, which reports high agreement with human judgments, strengthens the reliability of the experimental results.
Critical weaknesses
  • As I mentioned, my main concern is the limited novelty of the proposed methodology. Both region-level visual retrieval and iterative retrieval guided by evidence sufficiency have been extensively explored in prior works, including PixSearch, RegionRAG, and S2G-RAG, which are also cited in this paper. Although the authors combine these techniques for multimodal conversational memory, I do not see a substantial methodological difference from these earlier approaches beyond their integration into the proposed framework.
  • I also have concerns about the fairness of the baseline comparisons. COReMem uses several additional components, including SAM3, DINOv3, and multiple VLM calls for memory construction and iterative retrieval, whereas some baselines rely on simpler retrieval mechanisms. Although the authors provide retrieval-budget comparisons, it is unclear whether the improvements come from the proposed methodology itself or from using stronger visual encoders and additional computation. I believe more controlled comparisons with matched computational resources would strengthen the paper's claims (See my Questions For Authors)
Questions for authors
  1. How reliable are the VLM's binary evidence-sufficiency decisions? Every time I encounter model-based decisions being used, I have concerns about the reliability of the model judgments (especially when they play a crucial role in filtering/evaluating/labeling/routing like in this work). Can the authors, therefore, evaluate these judgments against oracle evidence labels and report false-positive and false-negative rates?
  2. COReMem uses DINOv3 ViT-7B for image-to-image retrieval, whereas MMRAG relies on SigLIP2-base. Could the authors evaluate MMRAG with the same DINOv3 encoder and retrieval budget as CoReMem? This would help clarify whether the reported improvements come from the proposed query-aware retrieval mechanism or simply from using a stronger visual representation.
  3. CoReMem performs multiple VLM calls for visual query construction and evidence-sufficiency checking, and can retrieve additional dialogue turns beyond the initial top-10. Could the authors compare CoReMem against an iterative MMRAG baseline that retrieves additional turns under the same total retrieval budget and computational cost (e.g., number of VLM calls or inference latency)?

Comments on the paper

1 comment

We sincerely thank the reviewers for their time and effort in providing thoughtful, detailed, and constructive feedback. We greatly appreciate their valuable insights and suggestions, which will help us further improve and strengthen our work. Although we have decided to withdraw this submission, we will carefully consider the feedback in our future revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.