acceptodds
Under review as a conference paper at ICLR 2027

BEYOND RANKING: MAKING LONG-TERM MULTI- MODAL MEMORY ADDRESSABLE

Abstract

Long-term multimodal memory is useful only if a future query can find the evidence it needs. Yet storing the right observation does not make it easy to retrieve, especially when evidence is visual, surrounded by similar memories, or distributed across multiple memory units. We call this problem the addressability gap. Across three long-term multimodal memory benchmarks, we study this gap through three retrieval challenges: finding image-bearing evidence, distinguishing the required observation from similar memories, and recovering the complete evidence set. We introduce CACE, an addressability-oriented framework with two components. At construction time, Contrastive Addressability Encoding (CAE) describes each observation in the context of visually similar memories, making its distinguishing details searchable while preserving the original observation as evidence. At retrieval time, Competitive Evidence Assignment (CEA) preserves individual query tokens and competitively assigns each token's matching credit across candidate memories, allowing different parts of the query to recruit different textual or visual evidence under a fixed retrieval budget. Experiments on MemEye, SMMBench, and Mem-Gallery show substantial improvements in evidence retrieval across all three challenges, with corresponding gains in downstream answer quality across multiple reader models. These results suggest that long-term multimodal memory should be designed not only to store relevant experience, but to make that experience addressable by future queries.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal memory, multimodal agent, retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_MnJmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies evidence retrieval from long-term multimodal memory. The authors first analyze three retrieval gaps, 1) finding image-bearing evidence, 2) distinguishing it from similar memories, and 3) recovering the complete evidence set. The proposed approach named CACE uses GPT-4.1 nano to caption each image together with its most similar images. The retrieval is done in two steps, first 30 candidates are obtained with a combination of text-to-image and text-to-text models. Second, the authors rerank the shortlist with ColQwen2, where the candidates compete for each query token through a softmax. Experiments on three benchmarks show consistent retrieval gains and better answers than SRAG-V in eight of nine settings.

Critical strengths

S1) The problem is relevant and well motivated. Evaluating the evidence retrieval separately from the answering is useful.

S2) The method is simple and training-free, and the reader receives the original observations instead of generated summaries.

S3) The retrieval improves significantly on all three benchmarks.

Critical weaknesses

W1) The ablations do not isolate the proposed contributions. The "w/o CEA" variant replaces the whole retrieval stage with SRAG-V (lines 429-431), removing the ColQwen2 reranker together with the competitive scoring, and "w/o CAE" removes the captions completely instead of only the neighbors. Without plain MaxSim on the same shortlist and captions generated without the neighbors, the gains could come from adding a much stronger model rather than from the proposed ideas. Comparisons to stronger retrievers such as GME [1] are also missing. Since CAE expands the index with captions and CEA reweights MaxSim, the methodological novelty is limited.

[1] Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models, Zhang et al., CVPR 2025

W2) The competitive scoring is a reweighted Chamfer similarity rather than an assignment of query parts to memories. Since the softmax denominator ZtZ_t is shared by all candidates, the score can be written as si=∑t1ZtStiexp⁡(Sti/ϵ)s_i = \sum_t \frac{1}{Z_t} S_{ti} \exp(S_{ti}/\epsilon), which is MaxSim with a shortlist-dependent token weight and a convex transform of the scores. The temperature ϵ=std(S)\epsilon = \mathrm{std}(S) (lines 310-311) is not justified or ablated, although it is a major part of the method. For ϵ→∞\epsilon \to \infty the reranking gives the same ranking as plain MaxSim, and for ϵ→0\epsilon \to 0 each token gives all its credit to the best candidate.

W3) The diagnostic study does not clearly separate the three gaps. An image-bearing unit may already contain the answer in its text, so the low ImgRecall does not show a specifically visual failure. Questions with and without similar neighbors are not compared at matching gold-set sizes, and completeness naturally drops with the number of required units even if the recall of each unit stays the same.

W4) It is not clear how the gold evidence was annotated and whether all gold units are needed to answer the question. Complete@10 requires the entire annotated set, so retrieving another observation with the same information counts as a failure. Questions with more than ten gold units can never be complete at K=10, but the gold-set sizes are not reported.

W5) Important details are missing. It is not mentioned whether ColQwen2 encodes the text and images of a memory unit jointly and in which order, which matters for its causal decoder. The corpus sizes are not reported and the method is not evaluated with an incrementally growing memory. Since the neighbors are searched in the whole collection at once (line 245), older memories are described using later observations.

Minor weaknesses:

MW1) The claim that stronger readers benefit more (lines 420-425) is not tested. With GPT-4.1 nano on MemEye, CACE is 0.4 points below SRAG-V despite the better retrieval.

MW2) SimpleMem-V is described as indexing the dataset captions in lines 117-119, but as searching LLM-generated summaries in lines 337-339. The authors should clarify which version is used in the experiments.

Questions for authors

Q1) How sensitive are the results to the temperature ϵ=std(S)\epsilon = \mathrm{std}(S) and shortlist size F?

Q2) How was gold evidence annotated? Are all listed units necessary, how is equivalent evidence treated, and how many questions require more than ten units?

Q3) How are text and multiple images encoded by ColQwen2, and how are captions maintained when new memories arrive?

Q4) What are the retrieval and answer-quality results with plain MaxSim on the same shortlist, and with captions generated by the same writer without neighbor images?

1 comment

I would like to thank the authors for the additional clarifications and experiments.

The new results show that MaxSim is much better baseline to re-ranking than the SRAG-V used in the paper. The proposed method beats it on MemEye by ~2%. So most of the gain comes from adding captions and the ColQwen2 reranker. Moreover, these performance gains only hold for the specific temperature parameter, with half or double the temperature it is on par with MaxSim. The performance also drops in shortlist size F, where we don't know about the MaxSim performance. It is important to compare to this baseline on other benchmarks as well. The same applies to the comparison of Ordinary vs Contrastive captions. In Complete@10 there is no difference between these two for MaxSim, and the difference for Competitive assignment is only 3/371 questions on the small MemEye benchmark. The gains of the two main modules CAE and CEA are thus hard to justify in the current version.

My other questions about the gold-set sizes and the ColQwen2 encoding are resolved.

I have carefully read the other reviews and the respective authors' answers as well. I will keep my initial score of weak reject.

Reviewer_tEXcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CACE, a method for retrieving evidence from long-term multimodal memory. It has two components: CAE, during memory construction, uses GPT-4.1 nano to describe each stored image in the context of visually similar images, aiming to make its distinctive details searchable; the second component is CEA, which, at query time, selects 30 candidate memories and uses ColQwen2 to match question tokens against their original text and images, assigning scores competitively across candidates. The authors report improved retrieval and, in most evaluated settings, better answer accuracy across three benchmarks.

Critical strengths
  • The paper addresses a practical problem in long-term multimodal memory.
  • The method is simple, training-free, query-independent, and keeps the original observations as evidence.
Critical weaknesses
  • The paper defines independent MaxSim scoring in Section 4.2 but does not report its performance. Comparing CEA with independent MaxSim ranking over the same 30 candidates, using the same ColQwen2 embeddings, is necessary to isolate the effect of competitive assignment.
  • Descriptions are produced by GPT-4.1 nano, and small models are typically weaker in tasks such as counting, labelling, and OCR, which are exactly the details CAE targets. The paper provides neither an analysis with a different or stronger captioner nor an assessment of caption quality (maybe using CLIP-S [1], PAC-S [2], LLM-judge). For example, hallucinations on a sample (e.g. CHAIR, POPE), rather than generic caption metrics that do not capture exact counts or labels.
  • The approach has limited novelty and positioning. The fine-grained token-to-evidence matching in CEA follows the MaxSim/late-interaction paradigm of ColBERT and multimodal retrievers such as ColPali. The apparent contribution is the cross-candidate softmax in Eq. 8 applied on top of these scores. The paper does not link this step to prior multimodal retrieval methods or to existing cross-candidate normalization schemes, and ColBERT and ColPali/ColQwen2 are not cited at all.
  • Also, the descriptions of the baselines are brief. SRAG-V is described clearly, but the other baselines (M2A, MMA) are described only briefly, and the paper does not compare their representations and retrieval procedures with CACE.
  • The paper lacks qualitative examples. A complete example showing stored memories, generated descriptions, a question, retrieved evidence before and after CACE, and the resulting answer would make the benchmark setting and the claimed retrieval improvements easier to assess. For someone unfamiliar with the datasets, it is difficult to follow the approach without dataset samples.

[1] CLIPScore: A Reference-free Evaluation Metric for Image Captioning.

[2] Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation

Questions for authors

See weaknesses, but in short:

  • How does CEA compare with independent MaxSim ranking over the same 30-candidate frontier, using the same ColQwen2 embeddings?
  • How does the cross-candidate softmax in Eq. 8 relate to prior multimodal late-interaction retrievers (ColBERT, ColPali/ColQwen2) and to existing cross-candidate normalization schemes? Could you add a table comparing CACE with these methods and with the baselines in terms of representation, retrieval procedure, and training requirements?
  • How do results change with a stronger captioner and with a different model family? How is the quality of the generated captions?
  • Could you add a complete example showing stored memories, the CAE description, a question, the retrieved evidence before and after CACE, and the final answer? Please include at least one failure case.
  • What fraction of gold evidence and of questions is actually visual in each benchmark, and how does a text-only retriever perform?
Reviewer_Yd17OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies evidence addressability in long-term multimodal memory: how future queries can retrieve relevant observations that have already been stored. The proposed CACE framework combines CAE, which generates distinctive indexing descriptions using existing similar images, with CEA, which applies competitive token-level scoring across candidate memories. Experiments evaluate evidence retrieval and downstream answering on three benchmarks.

Critical strengths

The problem is practically relevant. Evaluating evidence retrieval separately from final answer quality helps identify bottlenecks in multimodal memory systems.

Preserving original observations as evidence while using generated descriptions only for indexing is a sensible design choice.

The evaluation covers multiple benchmarks and readers, with substantial retrieval improvements on MemEye in particular.

Critical weaknesses

The experiments do not sufficiently isolate the contributions of the two central components. The “w/o CEA” variant replaces the retrieval stage with SRAG-V, removing both ColQwen2-based fine-grained matching and competitive scoring. It therefore remains unclear whether the gains come from stronger representations or the proposed scoring modification. A direct comparison with standard MaxSim, using identical candidates and ColQwen2 representations, is necessary. Similarly, the CAE ablation does not retain ordinary detailed descriptions generated by the same writer, leaving the benefits of additional searchable visual information and similar-image conditioning entangled.

The baseline comparisons are insufficiently targeted, and the robustness of the results needs further validation. Although several memory systems are evaluated, closely matched baselines that directly test the proposed design choices are missing. Improvements also vary considerably across benchmarks and metrics; for example, ImgRecall@10 improves by only 1.0 percentage point over SRAG-V on Mem-Gallery. Without confidence intervals, retrieval-level component ablations, and sensitivity analyses for candidate budgets and neighbor selection, the results do not yet provide sufficiently convincing support for the broader conclusions.

The additional costs are incompletely characterized. The paper usefully reports total execution time covering construction, retrieval, and answering. However, CAE introduces additional multimodal description-generation calls without separately reporting their input/output tokens, image-input usage, or API costs. CEA also adds multi-vector representation and reranking overhead. Consequently, the approximately 21% runtime increase does not establish a comparable increase in monetary cost or computational resources. One-time construction costs should be separated from per-query costs.

The writing and logical connections need improvement. The relationships between the three addressability gaps, the two proposed components, and the experimental conclusions are not always sufficiently clear. For example, improved retrieval on the similar-neighbor subset alone does not establish that the method reduces confusion caused by similar memories. The paper should more clearly distinguish problem diagnosis, design motivation, and experimental validation, and calibrate its mechanistic interpretations to the available evidence.

Questions for authors

How much does CEA improve over standard MaxSim with identical candidate sets and ColQwen2 representations?

How much does CAE improve over target-only descriptions generated by the same writer under comparable description budgets?

What additional token usage, image-input usage, and API costs does construction introduce, and how do these scale with memory size?

When new memories arrive, do existing neighbor sets and indexing descriptions require updating?

Reviewer_CYUdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies failures to retrieve stored multimodal evidence and proposes CACE. At construction time, CAE generates image descriptions conditioned on visually similar memories. At retrieval time, CEA assigns token-level matching credit competitively across candidates and selects a fixed number of memory units. Experiments on MemEye, SMMBench, and Mem-Gallery report improved evidence recall and completeness, alongside downstream gains in eight of nine benchmark-reader combinations.

Critical strengths

1.Evaluating evidence retrieval separately from answer quality is a useful choice. The complete-evidence metric captures failures that ordinary relevance scores can miss, and Table 4 shows improvements across all three benchmarks under the same ten-memory budget. 2.The design preserves original observations as evidence and uses generated descriptions only for indexing. Construction does not use evaluation questions, and one configuration is shared across benchmarks. The downstream results across several readers make the retrieval improvements more practically convincing.

Critical weaknesses

1.The ablations do not isolate the proposed mechanisms from simpler alternatives. Removing CAE also removes the generated descriptions, while removing CEA replaces ColQwen-based fine-grained scoring with coarse SRAG-V retrieval. Ordinary captions and independent late interaction using the same encoder and candidate frontier are needed to establish the value of contrastive description and competitive assignment specifically. 2.The runtime comparison is not sufficiently documented. Appendix E reports total wall-clock time, but does not specify hardware, API concurrency, caching, or the construction/retrieval/answering breakdown. These factors matter when comparing local multi-vector retrieval with API-driven memory systems, so the reported 21% overhead is difficult to interpret or reproduce.

Questions for authors

1.Are results available for ordinary image descriptions and standard independent late interaction, holding the writer, ColQwen encoder, and candidate frontier fixed? 2.What hardware, concurrency, and caching settings produced Table 6? Can the recorded total time be separated into construction, retrieval, and answering?

Comments on the paper

1 comment
Area_Chair_akTEOpenReview ↗

Please read the rebuttal and engage in discussion

Dear Reviewers,

The authors have posted their rebuttal. Please read it and engage in discussion, in particular:

  • Whether each of your questions and concerns has been addressed.
  • Whether your rating and recommendation have changed, and why.
  • Any remaining concerns that must be settled before the meta-review.

Thanks for your time and contribution.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.