- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper isolates how retrieved multimodal memories reach the answering model. DeliverMem preserves images, adds identifiers and timestamps, and incorporates region retrieval. Controlled experiments on MemLens and DMV-Bench examine these components and report improvements over published memory agents without training.
Critical strengths
The experiments hold retrieved evidence fixed while varying image content, timestamps, and image–message binding, exposing failures that retrieval recall misses.
The implementation is lightweight, and the analysis reports component-specific null effects alongside improvements.
Critical weaknesses
The headroom comparison depends on an uneven starting point. Delivery restores omitted images, while retrieval already finds at least one gold message on 96.5% of questions. A matched strong-caption baseline is absent. Oracle-retrieval gains also increase from 2.31 to 11.56 points between the 8B and 32B readers, suggesting that reader capability affects the estimated retrieval headroom.
The fixed interface introduces substantial task-specific regressions. Adding tags and dates improves overall MemLens accuracy by 1.03 points but reduces knowledge-update accuracy by 13.79 points. Across the full question set, the broader component combination also loses 8.62 points on knowledge updating. The optional fallback uses benchmark-provided question-type labels.
The tag experiments leave the addressability mechanism ambiguous. A failure trace naming the target object could simply repeat the query. Tags and accompanying instructions could also improve visual attention or recognition. The shuffled-tag control preserves positional cues through numbered text, and its measured effect is not statistically significant.
Questions for authors
Does the headroom ordering persist against strong image captions under matched retrieval?
Which component or interaction causes the knowledge-update regression?
Do tags improve selection when correct per-image object recognition has already been established?
3 comments
Response to Reviewer fTEu (Part 2/3)
Q2. Which component causes the knowledge-update regression?
The tag and the date together, mostly through the tag given back as the answer (revised Appendix E.8). With the tag, the model answers some knowledge-update questions ("which one did I switch to?") with the tag itself, "M2", instead of naming the item: on 30 of the 116 knowledge-update questions with tags and dates, 15 with tags alone, and none without tags; of the 14 it gets wrong where the configuration without tags and dates is right, 8 are such answers and none is an abstention. Each cue alone costs 2.59 there (n.s.); together they cost 8.62, the date roughly doubling how often the model points by tag. One sentence added to the instruction ("Use the tags to keep track of the items, but never answer with a tag: answer with what the item is.") cuts tag-only answers from 36 to 8 over the 789 questions. The sentence also raises accuracy without tags by +3.72 on the 699 answerable questions, through fewer abstentions, so we add it to both arms: the knowledge-update difference shrinks to −3.45 (5/9, p=0.42), and the temporal-reasoning gain stays (+9.79, p=0.0066). Every headline number stays on the configuration as submitted. The 789-question run retrieves on the region index in every arm, so the revision removes (R1) from the causes in E.8.
Q3. Tags once per-image recognition is established
On DMV-Bench we follow each image with a description written by the same 7B model from that image alone. Removing the tag then costs 3.23 on all 10 chains (1,888 probes, p=0.002): 1.1 where the description names the cue, so recognition is already in the text (6 of 10 chains, p=0.38), and 6.0 where it does not (10 of 10, p=0.002). The tag's effect sits where recognition from the text fails, which fits your reading that it helps the model find or recognize the right item as well as it fits addressing; these data do not separate the two. The revision describes (D2) accordingly in Section 5 and Appendix D.6. A separate control delivers no descriptions and compares the tag stamped in the image with the same tag in the text; there the stamp gains +9.3 on probes where the model can name the cue from the image alone (Part 1 of our reply to bS9q).
W2. The fallback uses question-type labels
Every headline number uses the single fixed configuration. The per-type selection rule gains +3.59 at 32K but reads the benchmark's question-type labels, which a deployed agent does not have (Appendix C.5).
W3. Positional cues and query echo
- Position. The shuffled-tag control is small (+2.63, p=0.072); the revision states the enumerated-text explanation as untested (Section 7). The controls that keep the enumeration but take away the distinct names are large: without descriptions, removing the tag costs 9.88 and identical tags cost 11.85 (both 10 of 10 chains, p=0.002). Both also change the instruction sentences that refer to the tags (Section 7; Appendices C.1 and C.3), so they do not separate the name from the instruction; in the 2×2 of Appendix C.3 the stamp alone, unmentioned in the text, gives +1.33 (n.s.), and the tag named in the text without a stamp gives +3.08.
- A trace that repeats the query. We agree: the worked traces do quote the instruction. The single-image descriptions above are written without it. The revised Section 4 notes that the trace can repeat the instruction's description of the cue, and Appendix D.6 states that it shows which item the model attributed the cue to, not that it recognized the cue.
The analysis of the tag after recognition (Appendix D.6) and the caveat on reasoning traces (Section 4) both started from your review, and the paper is clearer for them.
Revision summary (Part 3/3)
In short. All four reviews describe the paper as isolating the delivery stage, between retrieval and the answer, and that is what we set out to show. The revision keeps that claim and puts the evidence for it first.
- Delivery is a stage of its own: which parts of the retrieved memory reach the answering model, and in what form. Against no image, delivering the visual content of the retrieved messages adds 11.2 points more than perfect retrieval does with the gold messages alone, pooled over five answering models.
- The text-only published MemLens agents lose that content before their answering model. The benchmark's released pipeline puts an [image] marker in each image's place, a BLIP-2 caption recovers 2.3 of the 14.5 points the pixels add with the padded gold set in that run, and given the same 235B captions, Mem0, the strongest training-free agent, scores 27.75 on answerable questions and our pipeline, with no pixels at any stage, 40.46.
- Pixels and a caption written by a 235B model over every stored image are level at our deployed configuration; that caption takes 4,491 calls of the 235B model at 32K, while the pixels need no captioning call and cost more tokens at answer time. Neither form wins everywhere: a caption suffices when it holds what the question needs, which is unknown when it is written, and keeping the original, (D1), covers what it misses.
- DeliverMem is built from known parts (keep the original, give each item a readable identity, state when it was seen) and trains nothing; it leads the strongest published agent on MemLens at all four lengths and DMV-Bench's strongest method at every setting. On DMV-Bench, whose baseline already delivers the pixels, delivery with retrieval and images fixed adds 5 to 8 points at J = 5 to 15 on Qwen2.5-VL-7B.
- The stage leaves open questions that the decomposition makes measurable: which form to deliver for which question, where an item's identity should live (in the pixels it gains +7.89 and +8.11, in two runs, over the same tag in the text), and how a delivery cue interacts with the answer (a tag can itself be given as the answer; Q4 to bS9q, Q2 to fTEu).
Other changes. The Abstract is rewritten around these points; the Contributions put the stage first and DeliverMem as a deliberately simple instantiation; Figure 3 moves into Section 5; Section 4 adds the caption control (Table 2); "grows with backbone strength" is dropped; Section 6 adds Mem0 on our backbone; Appendix D.7 and E.12 are new, D.6 and E.8 are extended, and the prose is tightened throughout.
Thank you again for your time and careful reading. Your questions have made the paper's claim more precise, and we would be glad to clarify anything further during the discussion.
Response to Reviewer fTEu (Part 1/3): Q1 and W1, the headroom ordering against strong captions
Thank you for a clear and constructive review. You asked whether the headroom ordering holds against strong captions under matched retrieval; we ran that comparison on five answering models, and it is now in the revised Section 4 (Table 2; Appendix D.7).
The ordering depends on what delivery is measured against.
Against no image, close to what MemLens's text-only agents receive, delivery adds 11.21 points more than perfect retrieval does (p=0.002), pooled over five models with the gold messages alone.
Against a caption that a 235B model writes for every stored image, perfect retrieval adds +9.36, which is 7.05 points more than the pixels' +2.31 (p=0.010, gold messages alone), and the pixels match that caption (+2.31, p=0.18) without its write-time calls.
Against the answering model's own caption, the pixels lead with Table 1's retriever (+4.34, p=0.003, four models) and are level at our deployed configuration.
Setup (MemLens 32K, 173 answerable questions). Retrieved messages, answering model and prompt are fixed; only what sits in each image's place changes. Captions are written once per stored image without the question, by Qwen3-VL-235B-A22B or by the answering model with the same prompt. R* is Table 1's gold set, padded to ten messages ("with the gold messages alone" drops the padding; Table 16); R is Table 1's text-channel retriever; "deployed" is the configuration of Table 3. Pooled values average the models per question and test over questions (sign-flip permutation); the Pooled row gives each difference with its p in parentheses. The gold-only figures quoted above (+2.31, +9.36) are from Table 16; the table below uses R*.
* four models; the 235B model's own caption is the 235B caption.