- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies evidence retrieval from long-term multimodal memory. The authors first analyze three retrieval gaps, 1) finding image-bearing evidence, 2) distinguishing it from similar memories, and 3) recovering the complete evidence set. The proposed approach named CACE uses GPT-4.1 nano to caption each image together with its most similar images. The retrieval is done in two steps, first 30 candidates are obtained with a combination of text-to-image and text-to-text models. Second, the authors rerank the shortlist with ColQwen2, where the candidates compete for each query token through a softmax. Experiments on three benchmarks show consistent retrieval gains and better answers than SRAG-V in eight of nine settings.
Critical strengths
S1) The problem is relevant and well motivated. Evaluating the evidence retrieval separately from the answering is useful.
S2) The method is simple and training-free, and the reader receives the original observations instead of generated summaries.
S3) The retrieval improves significantly on all three benchmarks.
Critical weaknesses
W1) The ablations do not isolate the proposed contributions. The "w/o CEA" variant replaces the whole retrieval stage with SRAG-V (lines 429-431), removing the ColQwen2 reranker together with the competitive scoring, and "w/o CAE" removes the captions completely instead of only the neighbors. Without plain MaxSim on the same shortlist and captions generated without the neighbors, the gains could come from adding a much stronger model rather than from the proposed ideas. Comparisons to stronger retrievers such as GME [1] are also missing. Since CAE expands the index with captions and CEA reweights MaxSim, the methodological novelty is limited.
[1] Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models, Zhang et al., CVPR 2025
W2) The competitive scoring is a reweighted Chamfer similarity rather than an assignment of query parts to memories. Since the softmax denominator is shared by all candidates, the score can be written as , which is MaxSim with a shortlist-dependent token weight and a convex transform of the scores. The temperature (lines 310-311) is not justified or ablated, although it is a major part of the method. For the reranking gives the same ranking as plain MaxSim, and for each token gives all its credit to the best candidate.
W3) The diagnostic study does not clearly separate the three gaps. An image-bearing unit may already contain the answer in its text, so the low ImgRecall does not show a specifically visual failure. Questions with and without similar neighbors are not compared at matching gold-set sizes, and completeness naturally drops with the number of required units even if the recall of each unit stays the same.
W4) It is not clear how the gold evidence was annotated and whether all gold units are needed to answer the question. Complete@10 requires the entire annotated set, so retrieving another observation with the same information counts as a failure. Questions with more than ten gold units can never be complete at K=10, but the gold-set sizes are not reported.
W5) Important details are missing. It is not mentioned whether ColQwen2 encodes the text and images of a memory unit jointly and in which order, which matters for its causal decoder. The corpus sizes are not reported and the method is not evaluated with an incrementally growing memory. Since the neighbors are searched in the whole collection at once (line 245), older memories are described using later observations.
Minor weaknesses:
MW1) The claim that stronger readers benefit more (lines 420-425) is not tested. With GPT-4.1 nano on MemEye, CACE is 0.4 points below SRAG-V despite the better retrieval.
MW2) SimpleMem-V is described as indexing the dataset captions in lines 117-119, but as searching LLM-generated summaries in lines 337-339. The authors should clarify which version is used in the experiments.
Questions for authors
Q1) How sensitive are the results to the temperature and shortlist size F?
Q2) How was gold evidence annotated? Are all listed units necessary, how is equivalent evidence treated, and how many questions require more than ten units?
Q3) How are text and multiple images encoded by ColQwen2, and how are captions maintained when new memories arrive?
Q4) What are the retrieval and answer-quality results with plain MaxSim on the same shortlist, and with captions generated by the same writer without neighbor images?
I would like to thank the authors for the additional clarifications and experiments.
The new results show that MaxSim is much better baseline to re-ranking than the SRAG-V used in the paper. The proposed method beats it on MemEye by ~2%. So most of the gain comes from adding captions and the ColQwen2 reranker. Moreover, these performance gains only hold for the specific temperature parameter, with half or double the temperature it is on par with MaxSim. The performance also drops in shortlist size F, where we don't know about the MaxSim performance. It is important to compare to this baseline on other benchmarks as well. The same applies to the comparison of Ordinary vs Contrastive captions. In Complete@10 there is no difference between these two for MaxSim, and the difference for Competitive assignment is only 3/371 questions on the small MemEye benchmark. The gains of the two main modules CAE and CEA are thus hard to justify in the current version.
My other questions about the gold-set sizes and the ColQwen2 encoding are resolved.
I have carefully read the other reviews and the respective authors' answers as well. I will keep my initial score of weak reject.