acceptodds
Under review as a conference paper at ICLR 2027

Retrieved but Not Delivered: Multimodal Memory Delivery for Long-Term Agents

Abstract

Work on memory for multimodal agents optimizes what is written, updated and retrieved. Between retrieval and the answer, however, is a stage that multimodal memory evaluations do not isolate: what of the retrieved memory reaches the model, and in what form. We call it delivery, and a controlled decomposition on MemLens locates the remaining room there. With the retrieved evidence set exactly fixed, delivering the original pixels instead of withholding them raises accuracy by 13.87 points on an 8B backbone, whereas making retrieval perfect on those same messages improves it by 2.31. Delivery is the larger term on all three MemLens backbones and grows with backbone strength; retrieval grows too, without closing the gap. We propose DeliverMem, an instantiation of delivery as three decisions: keep the original modality, give each item a readable identity, and state when it was seen, with a retrieval-side adapter for the one property delivery cannot supply. Each is measured against a delivery-matched control that alters only its own variable. DeliverMem leads the strongest published memory agent on MemLens at all four context lengths, and beats DMV-Bench's own strongest method at every setting on both backbones. On MemLens it does this on a tenth to a seventieth of the input. Each decision helps only where the question lacks what it supplies, and is null elsewhere. A single fixed configuration nonetheless leads both benchmarks, without training any component or modifying the stored records.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal memory, long-term memory agents, memory delivery, vision-language models, retrieval-augmented agents, memory interfaces, controlled ablation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_fTEuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper isolates how retrieved multimodal memories reach the answering model. DeliverMem preserves images, adds identifiers and timestamps, and incorporates region retrieval. Controlled experiments on MemLens and DMV-Bench examine these components and report improvements over published memory agents without training.

Critical strengths

The experiments hold retrieved evidence fixed while varying image content, timestamps, and image–message binding, exposing failures that retrieval recall misses.

The implementation is lightweight, and the analysis reports component-specific null effects alongside improvements.

Critical weaknesses

The headroom comparison depends on an uneven starting point. Delivery restores omitted images, while retrieval already finds at least one gold message on 96.5% of questions. A matched strong-caption baseline is absent. Oracle-retrieval gains also increase from 2.31 to 11.56 points between the 8B and 32B readers, suggesting that reader capability affects the estimated retrieval headroom.

The fixed interface introduces substantial task-specific regressions. Adding tags and dates improves overall MemLens accuracy by 1.03 points but reduces knowledge-update accuracy by 13.79 points. Across the full question set, the broader component combination also loses 8.62 points on knowledge updating. The optional fallback uses benchmark-provided question-type labels.

The tag experiments leave the addressability mechanism ambiguous. A failure trace naming the target object could simply repeat the query. Tags and accompanying instructions could also improve visual attention or recognition. The shuffled-tag control preserves positional cues through numbered text, and its measured effect is not statistically significant.

Questions for authors

Does the headroom ordering persist against strong image captions under matched retrieval?

Which component or interaction causes the knowledge-update regression?

Do tags improve selection when correct per-image object recognition has already been established?

3 comments
AuthorsOpenReview ↗

Response to Reviewer fTEu (Part 1/3): Q1 and W1, the headroom ordering against strong captions

Thank you for a clear and constructive review. You asked whether the headroom ordering holds against strong captions under matched retrieval; we ran that comparison on five answering models, and it is now in the revised Section 4 (Table 2; Appendix D.7).

The ordering depends on what delivery is measured against.

Against no image, close to what MemLens's text-only agents receive, delivery adds 11.21 points more than perfect retrieval does (p=0.002), pooled over five models with the gold messages alone.

Against a caption that a 235B model writes for every stored image, perfect retrieval adds +9.36, which is 7.05 points more than the pixels' +2.31 (p=0.010, gold messages alone), and the pixels match that caption (+2.31, p=0.18) without its write-time calls.

Against the answering model's own caption, the pixels lead with Table 1's retriever (+4.34, p=0.003, four models) and are level at our deployed configuration.

Setup (MemLens 32K, 173 answerable questions). Retrieved messages, answering model and prompt are fixed; only what sits in each image's place changes. Captions are written once per stored image without the question, by Qwen3-VL-235B-A22B or by the answering model with the same prompt. R* is Table 1's gold set, padded to ten messages ("with the gold messages alone" drops the padding; Table 16); R is Table 1's text-channel retriever; "deployed" is the configuration of Table 3. Pooled values average the models per question and test over questions (sign-flip permutation); the Pooled row gives each difference with its p in parentheses. The gold-only figures quoted above (+2.31, +9.36) are from Table 16; the table below uses R*.

Answering model235B caption − none (R*)Pixels − 235B caption (R*)Pixels − 235B caption (deployed)Pixels − own caption (R)Pixels − own caption (deployed)ΔR (R* − R)
Qwen3-VL-8B+12.72+1.73+1.16−0.58+2.31+2.31
InternVL3.5-8B+13.87+2.31−2.31+3.47−4.05+9.83
Gemma-3-27B+15.030.00−2.89+6.94−4.62+8.09
Qwen3-VL-235B+15.03+2.89−0.58——+7.51
GLM-4.5V+20.81−0.58+2.31+7.51+2.89+1.73
Pooledall p<0.001+1.27 (0.47)−0.46 (0.85)+4.34 (0.003)*−0.87 (0.67)*+5.90 (0.007)

* four models; the 235B model's own caption is the 235B caption.

  • Uneven starting points. The two orderings above, against no image and against the 235B caption, use one ceiling, the gold messages alone, and one pooled test. Measured from these stronger starting points, both terms are small on Qwen3-VL-8B: perfect retrieval adds +2.31 over two-channel retrieval (Appendix D.5), and at the deployed configuration the pixels add +1.16 over the 235B caption.
  • What each form costs. The 235B caption takes 4,491 calls of that model at 32K (about 6.1M input and 0.6M output tokens), of which 9.4% are delivered for the benchmark's questions; a caption the system writes with its own model takes as many and is level with the pixels at the deployed configuration. The pixels need no captioning call at write time; their price is answer-time tokens (Q2 in our reply to 84yB).
  • When a caption is enough. On DMV-Bench, where a caption can be checked for the cue, our full delivery leads a 7B caption by 5.8 points on probes whose caption omits the cue and trails it by 4.1 where the caption names it (chain-level p = 0.057 and 0.012). Which details a later question needs is unknown at write time.
  • The retrieval term depends on the reader, as you note, and not monotonically in model size (ΔR, last column of the table; the 32B value is from the paper): +2.31 on Qwen3-VL-8B, +1.73 on GLM-4.5V, +7.5 to +9.8 on InternVL, Gemma and the 235B model, +11.56 on the 32B model. The revision drops "grows with backbone strength".
  • Second judge. GPT-5.4-mini agrees with the benchmark judge on 99.0% of 3,900 judgments; at the deployed configuration it gives −0.46 against the 235B caption and −1.16 against the own caption.
AuthorsOpenReview ↗

Response to Reviewer fTEu (Part 2/3)

Q2. Which component causes the knowledge-update regression?

The tag and the date together, mostly through the tag given back as the answer (revised Appendix E.8). With the tag, the model answers some knowledge-update questions ("which one did I switch to?") with the tag itself, "M2", instead of naming the item: on 30 of the 116 knowledge-update questions with tags and dates, 15 with tags alone, and none without tags; of the 14 it gets wrong where the configuration without tags and dates is right, 8 are such answers and none is an abstention. Each cue alone costs 2.59 there (n.s.); together they cost 8.62, the date roughly doubling how often the model points by tag. One sentence added to the instruction ("Use the tags to keep track of the items, but never answer with a tag: answer with what the item is.") cuts tag-only answers from 36 to 8 over the 789 questions. The sentence also raises accuracy without tags by +3.72 on the 699 answerable questions, through fewer abstentions, so we add it to both arms: the knowledge-update difference shrinks to −3.45 (5/9, p=0.42), and the temporal-reasoning gain stays (+9.79, p=0.0066). Every headline number stays on the configuration as submitted. The 789-question run retrieves on the region index in every arm, so the revision removes (R1) from the causes in E.8.

Q3. Tags once per-image recognition is established

On DMV-Bench we follow each image with a description written by the same 7B model from that image alone. Removing the tag then costs 3.23 on all 10 chains (1,888 probes, p=0.002): 1.1 where the description names the cue, so recognition is already in the text (6 of 10 chains, p=0.38), and 6.0 where it does not (10 of 10, p=0.002). The tag's effect sits where recognition from the text fails, which fits your reading that it helps the model find or recognize the right item as well as it fits addressing; these data do not separate the two. The revision describes (D2) accordingly in Section 5 and Appendix D.6. A separate control delivers no descriptions and compares the tag stamped in the image with the same tag in the text; there the stamp gains +9.3 on probes where the model can name the cue from the image alone (Part 1 of our reply to bS9q).

W2. The fallback uses question-type labels

Every headline number uses the single fixed configuration. The per-type selection rule gains +3.59 at 32K but reads the benchmark's question-type labels, which a deployed agent does not have (Appendix C.5).

W3. Positional cues and query echo

  • Position. The shuffled-tag control is small (+2.63, p=0.072); the revision states the enumerated-text explanation as untested (Section 7). The controls that keep the enumeration but take away the distinct names are large: without descriptions, removing the tag costs 9.88 and identical tags cost 11.85 (both 10 of 10 chains, p=0.002). Both also change the instruction sentences that refer to the tags (Section 7; Appendices C.1 and C.3), so they do not separate the name from the instruction; in the 2×2 of Appendix C.3 the stamp alone, unmentioned in the text, gives +1.33 (n.s.), and the tag named in the text without a stamp gives +3.08.
  • A trace that repeats the query. We agree: the worked traces do quote the instruction. The single-image descriptions above are written without it. The revised Section 4 notes that the trace can repeat the instruction's description of the cue, and Appendix D.6 states that it shows which item the model attributed the cue to, not that it recognized the cue.

The analysis of the tag after recognition (Appendix D.6) and the caveat on reasoning traces (Section 4) both started from your review, and the paper is clearer for them.

AuthorsOpenReview ↗

Revision summary (Part 3/3)

In short. All four reviews describe the paper as isolating the delivery stage, between retrieval and the answer, and that is what we set out to show. The revision keeps that claim and puts the evidence for it first.

  1. Delivery is a stage of its own: which parts of the retrieved memory reach the answering model, and in what form. Against no image, delivering the visual content of the retrieved messages adds 11.2 points more than perfect retrieval does with the gold messages alone, pooled over five answering models.
  2. The text-only published MemLens agents lose that content before their answering model. The benchmark's released pipeline puts an [image] marker in each image's place, a BLIP-2 caption recovers 2.3 of the 14.5 points the pixels add with the padded gold set in that run, and given the same 235B captions, Mem0, the strongest training-free agent, scores 27.75 on answerable questions and our pipeline, with no pixels at any stage, 40.46.
  3. Pixels and a caption written by a 235B model over every stored image are level at our deployed configuration; that caption takes 4,491 calls of the 235B model at 32K, while the pixels need no captioning call and cost more tokens at answer time. Neither form wins everywhere: a caption suffices when it holds what the question needs, which is unknown when it is written, and keeping the original, (D1), covers what it misses.
  4. DeliverMem is built from known parts (keep the original, give each item a readable identity, state when it was seen) and trains nothing; it leads the strongest published agent on MemLens at all four lengths and DMV-Bench's strongest method at every setting. On DMV-Bench, whose baseline already delivers the pixels, delivery with retrieval and images fixed adds 5 to 8 points at J = 5 to 15 on Qwen2.5-VL-7B.
  5. The stage leaves open questions that the decomposition makes measurable: which form to deliver for which question, where an item's identity should live (in the pixels it gains +7.89 and +8.11, in two runs, over the same tag in the text), and how a delivery cue interacts with the answer (a tag can itself be given as the answer; Q4 to bS9q, Q2 to fTEu).

Other changes. The Abstract is rewritten around these points; the Contributions put the stage first and DeliverMem as a deliberately simple instantiation; Figure 3 moves into Section 5; Section 4 adds the caption control (Table 2); "grows with backbone strength" is dropped; Section 6 adds Mem0 on our backbone; Appendix D.7 and E.12 are new, D.6 and E.8 are extended, and the prose is tightened throughout.

Thank you again for your time and careful reading. Your questions have made the paper's claim more precise, and we would be glad to clarify anything further during the discussion.

Reviewer_bS9qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies memory delivery as an overlooked bottleneck in multimodal long-term agents, arguing that retrieving relevant memories is not enough if their content is not presented to the model in a usable form. Through controlled experiments, the authors show that preserving original visual evidence can improve accuracy substantially more than perfecting retrieval. They propose DeliverMem, a training-free framework that preserves original modalities, assigns readable identifiers to retrieved items, and provides temporal information, complemented by a region-aware retrieval mechanism. Evaluations on MemLens and DMV-Bench demonstrate consistent improvements over prior memory agents across multiple settings, often with substantially less input context. Further ablations isolate the contributions of individual components and show that their benefits depend on the information required by each task. The paper's central contribution is to establish memory delivery as a distinct and consequential stage in multimodal agent memory systems.

Critical strengths
  1. Novel and well-motivated perspective. The paper identifies memory delivery as an overlooked bottleneck and uses controlled experiments to demonstrate its importance relative to retrieval.
  2. Strong empirical evidence. DeliverMem consistently outperforms prior memory agents on two benchmarks without additional training, while carefully designed ablations substantiate the effectiveness of its components.
Critical weaknesses
  1. Limited novelty and potentially overstated central insight. The main delivery gain largely comes from restoring raw images for questions that explicitly require visual evidence, an observation already suggested by prior multimodal benchmarks. Moreover, the delivery–retrieval comparison is dominated by information-extraction questions and relies on a retrieval baseline that already finds the relevant evidence in most cases. It remains unclear whether the paper establishes a genuinely new bottleneck beyond the well-known importance of preserving task-relevant information.
  2. Limited incremental contribution and generalizability. DeliverMem largely combines established techniques, while its additional delivery mechanisms provide modest gains once raw images are preserved (+1.03 points on MemLens 32K). Much of the improvement on DMV-Bench comes from region-aware retrieval rather than delivery. Component effectiveness is highly task-dependent, sometimes degrading performance, and the claimed advantage of delivery over retrieval is statistically significant only on the 8B backbone. These findings weaken the case for DeliverMem as a broadly effective, novel framework.
Questions for authors
  1. Beyond preserving original images, what is the statistically significant contribution of the proposed delivery mechanisms over a strong, image-preserving baseline with matched retrieval quality, backbone, and input budget?
  2. Given that the delivery gains are heavily concentrated in information-extraction questions, how do the authors distinguish a general memory-delivery bottleneck from the expected benefit of providing visual evidence for visually grounded questions?
  3. Since the oracle retrieval condition truncates gold messages to a fixed budget and pads shorter sets with additional messages, to what extent does it represent the true retrieval ceiling? Would the delivery–retrieval comparison hold under stronger retrieval strategies or alternative evidence-selection budgets?
  4. What explains the substantial degradation on knowledge-update questions and the inconsistent benefits across tasks and backbones? How do these results support the use of a single fixed delivery configuration?
  5. Given that the delivery–retrieval gap is not statistically significant on the two larger backbones, what evidence supports the claim that delivery is generally a more important bottleneck as model capabilities improve?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bS9q (Part 1/3): Q2 and W1, delivery with the visual content held fixed

Thank you for the thoughtful review, and for questions precise enough that one of them, Q3, corrected a statement of ours (Part 2). Your Q2 goes to the heart of the paper, and we take it first: how a general delivery bottleneck differs from "the expected benefit of providing visual evidence for visually grounded questions".

Delivery does more than restore the image.

With the same images and the same messages, on DMV-Bench the item tags and viewing-order notes add 5 to 8 points at J = 5 to 15 on Qwen2.5-VL-7B (with our retriever, more than half of DeliverMem's margin over DualMem); on MemLens, stating when each item was seen adds +8.76 on temporal-reasoning questions over giving no date (28/11), and keeping each image after its own message adds +6.36 over shuffling them (14/3). The image term itself, concentrated on information extraction as you note, is what the text-only published agents lose before their answering model.

DMV-Bench, delivery against the same retrieval with the same images and no tag or [seen #k of N] note (mean over chains; chain-level exact permutation p; ten chains on Qwen2.5-VL-7B, six on Gemini 2.5 Flash; revised Table 23, Appendix E.12):

J=5J=10J=15J=50
Qwen2.5-VL-7B, DualMem's retriever+5.22 (0.045)+6.71 (0.002)+5.65 (0.004)+1.39 (0.25)
Qwen2.5-VL-7B, (R1)+8.25 (0.035)+7.81 (0.012)+7.76 (0.008)+6.53 (0.004)
Gemini 2.5 Flash, DualMem's retriever–+0.05 (0.94)–+0.31 (0.44)
Gemini 2.5 Flash, (R1)–+0.49 (0.16)–+0.61 (0.13)

Where an item's identity lives, on DMV-Bench with Qwen2.5-VL-7B at J=10 (1,888 probes):

  • In the pixels: with the pixels fixed, moving each item's tag from the text into its image gains +8.11 averaged over chains (9 of 10 chains, p=0.0098; +7.89 in the submitted run; Appendix C.3), both where the model can name the cue from that image alone (+9.3 over those probes) and where it cannot (+7.4).
  • The tag after recognition: with a one-line description under each image, written by the same 7B model from that image alone, removing the tag costs 3.23 (10 of 10 chains, p=0.002), mostly where the description does not name the cue (6.0, against 1.1 where it does, n.s.).

On MemLens (Table 2; full table in Part 1 of our replies to 84yB, gkaV and fTEu; the padded gold set is Table 1's gold set padded to ten messages, "the gold messages alone" drops the padding, and the deployed configuration is Table 3's), the text-only published agents lose the visual content before their answering model: the released pipeline puts an [image] marker in each image's place, a BLIP-2 caption recovers 2.31 of the 14.45 points the pixels add with the padded gold set in that run (13.87 in Table 1's run), and Mem0 with the same 235B captions scores 27.75 on answerable questions against 40.46 for our pipeline with no pixels at any stage. A 235B caption written over every stored image carries most of that content; against it the pixels are level at the deployed configuration without its 4,491 write-time calls, and with the gold messages alone perfect retrieval adds +9.36, which is 7.05 points more than the pixels' +2.31 (p=0.010). Prior benchmarks, including MemEye, which we cite, show that visual evidence helps visual questions; they do not separate delivery from retrieval on the same evidence or show where the published agents lose it.

AuthorsOpenReview ↗

Response to Reviewer bS9q (Part 2/3)

Q1. What delivery adds over an image-preserving baseline

On DMV-Bench it is significant on Qwen2.5-VL-7B at every J with (R1) and at J = 5 to 15 with DualMem's retriever, and under one point on Gemini 2.5 Flash (table in Part 1), where (R1) carries most of the margin over DualMem. On MemLens, DeliverMem against the same retrieval with images and no tags or dates is +1.03 on the 195-question split and −1.72 on the 699 answerable questions of the full set (both n.s.). By type on the full set: temporal reasoning +7.22 (p=0.038), knowledge update −8.62 (p=0.031; Q4), information extraction −4.88 (4/16, p=0.012), multi-session −2.80 (n.s.).

Q3. Is perfect retrieval a true ceiling?

You are right that the padded oracle understates the ceiling, and we have corrected our statement. The gold-only condition gives all gold messages, without truncation and without padding; removing the padding gains +4.05 (11/4, Appendix D.5). On Qwen3-VL-8B the ceiling is therefore +6.36 over Table 1's text-channel retriever (2.31 + 4.05) and +2.31 over our two-channel retriever, which adds an image channel (18/14). Pooled over five models, perfect retrieval to this ceiling adds +9.36 and delivery against no image +20.58, so D − R is +11.21 (p=0.002). Raising the budget from K=10 to K=20 leaves DeliverMem's 32K accuracy at 51.28 (Appendix E.5). The revised Section 4 reports both ceilings, with the pooled contrasts in Table 16 (Appendix D.7).

Q4. Knowledge update and the single fixed configuration

The tag and the date together, mostly through the tag being given as the answer (revised Appendix E.8). With the tag, the model answers some knowledge-update questions ("which one did I switch to?") with the tag itself, "M2". On all 789 questions this happens on 30 of the 116 knowledge-update questions with tags and dates, 15 with tags alone, and none without tags; of the 14 it gets wrong where the configuration without tags and dates is right, 8 are such answers. Each cue alone costs 2.59 there (n.s.); together they cost 8.62. One sentence added to the instruction ("Use the tags to keep track of the items, but never answer with a tag: answer with what the item is.") cuts tag-only answers over the 789 questions from 36 to 8. The sentence also raises accuracy without tags by +3.72 on the 699 answerable questions, through fewer abstentions, so we add it to both arms: the knowledge-update difference shrinks to −3.45 (5/9, p=0.42), and the temporal-reasoning gain stays (+9.79, p=0.0066). Every headline number stays on the configuration as submitted. The per-type selection rule gains +3.59 at 32K but reads the benchmark's question-type labels, which a deployed agent does not have (Appendix C.5). The 789-question run retrieves on the region index in every arm, so the revision removes (R1) from the causes in E.8.

Q5. Is delivery the larger bottleneck as models improve?

Against no image it holds pooled over five answering models: with the gold messages alone, D − R (the delivery term minus the retrieval term) is +11.21 (p=0.002), positive on each model and significant on three after Holm; with the padded gold set it is positive on all seven answering models and significant on two after Holm. Against a 235B caption the order reverses: perfect retrieval adds 7.05 points more than the pixels do (Part 1). The revised text drops "grows with backbone strength" from the Abstract, Introduction and Section 4.

Your Q3 and Q5 changed how the main claim is stated: the Abstract, Introduction and Section 4 now use the gold-only ceiling and rest on the pooled test over five answering models rather than a trend across backbones. We are grateful for both.

AuthorsOpenReview ↗

Revision summary (Part 3/3)

In short. All four reviews describe the paper as isolating the delivery stage, between retrieval and the answer, and that is what we set out to show. The revision keeps that claim and puts the evidence for it first.

  1. Delivery is a stage of its own: which parts of the retrieved memory reach the answering model, and in what form. Against no image, delivering the visual content of the retrieved messages adds 11.2 points more than perfect retrieval does with the gold messages alone, pooled over five answering models.
  2. The text-only published MemLens agents lose that content before their answering model. The benchmark's released pipeline puts an [image] marker in each image's place, a BLIP-2 caption recovers 2.3 of the 14.5 points the pixels add with the padded gold set in that run, and given the same 235B captions, Mem0, the strongest training-free agent, scores 27.75 on answerable questions and our pipeline, with no pixels at any stage, 40.46.
  3. Pixels and a caption written by a 235B model over every stored image are level at our deployed configuration; that caption takes 4,491 calls of the 235B model at 32K, while the pixels need no captioning call and cost more tokens at answer time. Neither form wins everywhere: a caption suffices when it holds what the question needs, which is unknown when it is written, and keeping the original, (D1), covers what it misses.
  4. DeliverMem is built from known parts (keep the original, give each item a readable identity, state when it was seen) and trains nothing; it leads the strongest published agent on MemLens at all four lengths and DMV-Bench's strongest method at every setting. On DMV-Bench, whose baseline already delivers the pixels, delivery with retrieval and images fixed adds 5 to 8 points at J = 5 to 15 on Qwen2.5-VL-7B.
  5. The stage leaves open questions that the decomposition makes measurable: which form to deliver for which question, where an item's identity should live (in the pixels it gains +7.89 and +8.11, in two runs, over the same tag in the text), and how a delivery cue interacts with the answer (a tag can itself be given as the answer; Q4 to bS9q, Q2 to fTEu).

Other changes. The Abstract is rewritten around these points; the Contributions put the stage first and DeliverMem as a deliberately simple instantiation; Figure 3 moves into Section 5; Section 4 adds the caption control (Table 2); "grows with backbone strength" is dropped; Section 6 adds Mem0 on our backbone; Appendix D.7 and E.12 are new, D.6 and E.8 are extended, and the prose is tightened throughout.

Thank you again for your time and careful reading. Your questions have made the paper's claim more precise, and we would be glad to clarify anything further during the discussion.

Reviewer_gkaVOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the delivery step in multimodal long term memory, namely the form in which retrieved records are passed to the answer model. Using an oracle retriever on MemLens, it decomposes accuracy into a delivery term and a retrieval term. With Qwen3-VL-8B, delivering pixels adds 13.87 points while oracle retrieval adds only 2.31 points. Based on this finding, the paper proposes DeliverMem, a training free framework. It keeps the original images, renders identity labels such as M1 and M2 into the pixels, and attaches dates or order indices. It also adds a region multi vector retriever that scores a whole image view and a two by two overlapping grid with MaxSim. DeliverMem reaches 51.28 on MemLens at 32K against 32.82 for the strongest published agent, and leads DualMem in all eight DMV-Bench settings.

Critical strengths
  1. The paper isolates delivery as an independent stage. Equation 1 fixes the retrieved set and changes one factor at a time, so the delivery term and the retrieval term are measured from the same oracle corner.
  2. Each component is paired with a delivery matched control. The counterfactual image ladder in Table 12 shows that a decoy photo and no photo give the same loss of 15.03 points, so the gain comes from image content rather than the presence of image tokens. The label study in Table 10 shows that labels in pixels alone give 1.33 points and labels in text alone give 3.08 points, while both together give 10.97 points. This separates the uniqueness of an identifier from its salience.
  3. The statistics fit the data structure. Paired sign tests are run on discordant pairs, and on DMV-Bench the exact cluster permutation test treats chains rather than nested probes as units. Null results such as R1 on MemLens and D3 on DMV-Bench are reported rather than hidden.
Critical weaknesses
  1. The text only baseline D0 removes the pixels and keeps only the original message text. It is not a caption proxy. The caption at write time systems criticized in Section 2, such as MIRIX and MemVerse, still pass captions to the answer model. So the 13.87 point delivery term in Table 1 mainly compares an image with no image. It does not show that pixels beat a good textual proxy. I suggest the authors recompute the delivery term against strong captions, for example from Qwen3-VL-235B-A22B, and against question aware captions.
  2. The main conclusion rests on a narrow subset and one model family. Table 13 shows that the delivery gain is 40.98 points on the 61 information extraction questions, 2.86 points on multi session reasoning, and zero on temporal reasoning and knowledge update. Table 15 shows that the ratio of the delivery term to the retrieval term drops from 6.00 on 8B to 1.55 on 32B, and the difference between the two terms is no longer significant on the larger models. All MemLens answer models and the judge come from the Qwen3-VL family. It is unclear whether the claim holds on heterogeneous backbones such as InternVL or GPT models, given that delivery is also null for Gemini 2.5 Flash on DMV-Bench in Table 17.
  3. The proposed components add little on MemLens. R1 is null at minus 1.03 points, D2 against constant labels gives 0.51 points, and labels plus dates add only 1.03 points in Appendix E.8. The full configuration even loses 8.62 points on knowledge update. In essence D2 is Set-of-Mark applied to retrieved items, R1 is ColBERT style MaxSim over image crops, and D3 is a standard timestamp. What remains effective on MemLens is query conditioned retrieval with images kept, which is a natural multimodal RAG baseline. The paper should clarify what is new at the component level beyond the analysis framework.
  4. The comparison with published agents is confounded. Table 7 shows that most published agents use 2B to 7B text models or older vision language models, and only M2A shares the backbone. Of the 36 question gap at 32K, 19 come from abstention questions, where DeliverMem reaches 100 percent and MemAgent-7B reaches 13.64. On answerable questions the gap shrinks to 9.83 points. I suggest rerunning Mem0 and MemAgent with the same backbone and reporting answerable and abstention questions separately.
Questions for authors
  1. How large is the delivery term when D0 uses strong captions? If pixels still clearly beat strong captions, I would consider raising my score.
  2. Does the decomposition in Table 1 hold with a stronger multimodal retriever and with non Qwen answer models and judges?
  3. Shuffling the binding between images and messages costs 6.36 points in Table 19, yet binding is not part of DeliverMem. Please explain this choice.
3 comments
AuthorsOpenReview ↗

Response to Reviewer gkaV (Part 1/3): Q1, W1 and W2, strong and question-aware captions

Thank you for a detailed and expert review. We followed your suggestions closely: both caption conditions you proposed, a 235B caption and a question-aware one, are now measured on five answering models (revised Section 4, Table 2; Appendix D.7).

To answer your first question directly: the pixels do not clearly beat a 235B caption, but they match it without its 4,491 write-time calls.

They lead it by +2.31 with the gold messages alone (p=0.18) and are level at the deployed configuration (−0.46, p=0.85), and over that caption, with the gold messages alone, perfect retrieval adds +9.36, which is 7.05 points more than the pixels' +2.31 (p=0.010).

The paper's claim is about the delivery stage itself: whether the visual content of the retrieved messages reaches the answering model decides accuracy, and the text-only published agents lose it before their answering model. Against no image, with the gold messages alone, delivery adds 11.21 points more than perfect retrieval does (p=0.002; five models).

Setup (MemLens 32K, 173 answerable questions). Retrieved messages, answering model and prompt are fixed; only what sits in each image's place changes. Captions are written once per stored image without the question, by Qwen3-VL-235B-A22B or by the answering model with the same prompt. R* is Table 1's gold set, padded to ten messages ("with the gold messages alone" drops the padding; Table 16); R is Table 1's text-channel retriever; "deployed" is the configuration of Table 3. Pooled values average the models per question and test over questions (sign-flip permutation); the Pooled row gives each difference with its p in parentheses. The gold-only figures quoted above (+2.31, +9.36) are from Table 16; the table below uses R*.

Answering model235B caption − none (R*)Pixels − 235B caption (R*)Pixels − 235B caption (deployed)Pixels − own caption (R)Pixels − own caption (deployed)ΔR (R* − R)
Qwen3-VL-8B+12.72+1.73+1.16−0.58+2.31+2.31
InternVL3.5-8B+13.87+2.31−2.31+3.47−4.05+9.83
Gemma-3-27B+15.030.00−2.89+6.94−4.62+8.09
Qwen3-VL-235B+15.03+2.89−0.58——+7.51
GLM-4.5V+20.81−0.58+2.31+7.51+2.89+1.73
Pooledall p<0.001+1.27 (0.47)−0.46 (0.85)+4.34 (0.003)*−0.87 (0.67)*+5.90 (0.007)

* four models; the 235B model's own caption is the 235B caption.

  • Where the text-only agents lose it. MemLens's released pipeline puts an [image] marker in each image's place; a BLIP-2 caption recovers 2.31 of the 14.45 points the pixels add with the padded gold set in that run (Appendix D.7); W4 in Part 2 compares Mem0 and our pipeline with your 235B captions.
  • Where the delivery term sits (your W2). On the 61 information-extraction questions, whose answer is a fact in one image, delivery adds +40.98 and perfect retrieval −1.64 (Appendices D.4–D.5). On them DeliverMem scores 55.74 and no published agent exceeds 18.03, so they carry our lead (Section 6).
  • Captions in practice. The write-time captioning systems you cite caption with the model that answers (MIRIX: Gemini 2.5 Flash throughout) or a small one (MemVerse: GPT-4o-mini): the "own caption" column. The pixels add +4.34 (p=0.003) with Table 1's retriever and are level at the deployed configuration (−0.87, p=0.67).
  • What each form costs. The 235B caption takes 4,491 calls of that model at 32K (about 6.1M input and 0.6M output tokens), of which 9.4% are delivered for the benchmark's questions; an own caption takes as many. The pixels need no captioning call at write time; their price is answer-time tokens.
  • Question-aware captions (235B, question in view), pooled over five models with the padded gold set: over the question-blind caption, +3.35 overall (n.s.), +16.07 on information extraction (p=0.020) and −3.57 on the other types (p=0.030); question-aware caption minus pixels, +2.08 (n.s.). With Table 1's retriever: +14.10 on information extraction, −2.32 on the other types, and −0.69 against the pixels (n.s.). This agrees with PromptCap (Hu et al., ICCV 2023). Such a caption can only be written if the original is kept until the question arrives, which is what (D1) does.
  • Judge. GPT-5.4-mini agrees with the benchmark judge on 99.0% of 3,900 judgments; at the deployed configuration it gives −0.46 against the 235B caption and −1.16 against the own caption.
AuthorsOpenReview ↗

Response to Reviewer gkaV (Part 2/3)

Q2. Non-Qwen readers and judges; a stronger retriever

Against no image, delivery adds more than perfect retrieval in point estimate on every answering model we tried, three of them outside the Qwen family. D − R, which reduces to A(R, pixels) − A(R*, no image): Qwen3-VL-8B +11.56 (p=0.0078), 30B-A3B +5.78 (p=0.18), 32B +6.36 (p=0.14) (these three in the paper), InternVL +6.36 (p=0.16), Gemma +6.94 (p=0.14), Qwen3-VL-235B +10.40 (p=0.011), GLM-4.5V +18.50 (p=9e-6). After Holm across the seven, 8B and GLM remain significant. With the gold messages alone, pooled over five models, it is +11.21 (p=0.002). The second judge is in Part 1. Against a 235B caption the order reverses: perfect retrieval adds 7.05 points more than the pixels do (Part 1).

On a stronger retriever: our two-channel retriever, which adds an image channel to the text one, comes within 2.31 points of the gold-only ceiling on Qwen3-VL-8B, while Table 1's text-channel retriever is 6.36 below it (2.31 from the padded oracle and 4.05 from removing the padding; Appendix D.5). The gold-only condition delivers the gold messages without padding or distractors.

Q3. Image–message binding

You are right, and thank you for catching it. Every configuration, including the deployed one, already places each image after its own message; the shuffle removes exactly that (Appendix C.1). The revision relabels the row "Binding (not deployed)" of Table 6 (Table 5 in the submission) as "Binding (kept by construction)" and corrects the Conclusion. It is a measured property of delivery (+6.36, 14/3, p=0.013), present in any pipeline that keeps images in place.

W3. Novelty of the components

The primitives are indeed known, and we cite them: (D2) relocates Set-of-Mark to the items of a retrieved set and (R1) is MaxSim over whole and regional encodings. The paper's contribution is the stage they act on and the evidence for when each helps. The "natural multimodal RAG baseline" is the paper's own text-channel row, 48.72 at 32K, which no published MemLens agent reaches (best 32.82): delivering the retrieved messages with their visual content is the step those agents miss. Beyond it:

  • On DMV-Bench, whose baseline already delivers pixels, delivery with the retrieved items and images fixed adds +5.2 to +8.3 points at J = 5 to 15 on Qwen2.5-VL-7B, significant at each (Table 23; on Gemini 2.5 Flash under one point, where (R1) carries most of the margin over DualMem).
  • Tag placement, with the images kept: moving each item's tag from the text into its image gains +8.11 averaged over chains (89.67 vs. 81.20 over all probes; 9 of 10 chains, p=0.0098; +7.89 in the submitted run, Appendix C.3).
  • Captions instead of images: a caption with the tag in its text, delivered without the image, scores 89.46, level with our full delivery (89.67), which needs no captioning call. It beats the tag written in the text beside the pixels (81.20), and that lead comes from probes whose caption names the cue.
  • On MemLens the date carries the temporal questions (+13.92 against permuted dates, p=3.5e-6, Table 5), and the knowledge-update cost you note comes mostly from the model answering with the tag; one instruction sentence shrinks it to −3.45 (n.s.) while the temporal gain stays (Q4 of our reply to bS9q).

W4. Published agents

  • MemAgent-7B, Memory-T1 and M3-Agent learn their memory policy by reinforcement learning on their own backbone; a same-backbone rerun would mean repeating that training, which is beyond a discussion-period experiment, and their rows are MemLens's published results. DeliverMem trains nothing and leads every published agent at all four lengths (Table 3).
  • Mem0 and MemOS are training-free, like DeliverMem, and Mem0 is the strongest of them at 32K and the best published agent at 64K and 256K. Rerun with MemLens's released pipeline on our Qwen3-VL-8B, it scores 32.31 (28.32 answerable), against its published 31.79 (26.01).
  • With the 235B caption in place of each image marker of that pipeline, Mem0 scores 27.75 on answerable questions; DeliverMem with the same captions delivered in place of its images, its image channel still retrieving on the stored pixels, scores 42.77 (+15.03, 42/16, p=0.0009), and 40.46 with no pixels at any stage (text-channel retrieval, captions delivered). With the answering model's own captions, Mem0 29.48, DeliverMem 41.62 (+12.14, p=0.013). Mem0 passes its extracted memories, not the messages, to the answering model.
  • Answerable and abstention accuracy for every system are in Table 8; Section 6 gives the margin on answerable questions (+9.83).

Thanks to your suggestions, the revision now has the 235B-caption columns of Table 2 and the question-aware captions of Appendix D.7, answering models and a judge outside the Qwen family, Mem0 on our backbone (Section 6), and a corrected binding row and Conclusion.

AuthorsOpenReview ↗

Revision summary (Part 3/3)

In short. All four reviews describe the paper as isolating the delivery stage, between retrieval and the answer, and that is what we set out to show. The revision keeps that claim and puts the evidence for it first.

  1. Delivery is a stage of its own: which parts of the retrieved memory reach the answering model, and in what form. Against no image, delivering the visual content of the retrieved messages adds 11.2 points more than perfect retrieval does with the gold messages alone, pooled over five answering models.
  2. The text-only published MemLens agents lose that content before their answering model. The benchmark's released pipeline puts an [image] marker in each image's place, a BLIP-2 caption recovers 2.3 of the 14.5 points the pixels add with the padded gold set in that run, and given the same 235B captions, Mem0, the strongest training-free agent, scores 27.75 on answerable questions and our pipeline, with no pixels at any stage, 40.46.
  3. Pixels and a caption written by a 235B model over every stored image are level at our deployed configuration; that caption takes 4,491 calls of the 235B model at 32K, while the pixels need no captioning call and cost more tokens at answer time. Neither form wins everywhere: a caption suffices when it holds what the question needs, which is unknown when it is written, and keeping the original, (D1), covers what it misses.
  4. DeliverMem is built from known parts (keep the original, give each item a readable identity, state when it was seen) and trains nothing; it leads the strongest published agent on MemLens at all four lengths and DMV-Bench's strongest method at every setting. On DMV-Bench, whose baseline already delivers the pixels, delivery with retrieval and images fixed adds 5 to 8 points at J = 5 to 15 on Qwen2.5-VL-7B.
  5. The stage leaves open questions that the decomposition makes measurable: which form to deliver for which question, where an item's identity should live (in the pixels it gains +7.89 and +8.11, in two runs, over the same tag in the text), and how a delivery cue interacts with the answer (a tag can itself be given as the answer; Q4 to bS9q, Q2 to fTEu).

Other changes. The Abstract is rewritten around these points; the Contributions put the stage first and DeliverMem as a deliberately simple instantiation; Figure 3 moves into Section 5; Section 4 adds the caption control (Table 2); "grows with backbone strength" is dropped; Section 6 adds Mem0 on our backbone; Appendix D.7 and E.12 are new, D.6 and E.8 are extended, and the prose is tightened throughout.

Thank you again for your time and careful reading. Your questions have made the paper's claim more precise, and we would be glad to clarify anything further during the discussion.

Reviewer_84yBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the delivery stage of agent memory, which sits between retrieval and answering: what of the retrieved memory reaches the model, and in what form. The authors propose DeliverMem, which keeps the original images (D1), stamps a readable identifier on each item (D2), states when each item was seen (D3), and adds a region-level multi-vector retriever (R1). On both MemLens and DMV-Bench, DeliverMem outperforms existing memory agents without any additional training.

Critical strengths
  1. I find the separation of delivery from retrieval a useful perspective. The 2 x 2 decomposition of A(R, D) is intuitive.

  2. The experiments and statistical analyses are rigorous. The ladder of counterfactual images shows that the gain comes from image content rather than from extra image tokens. The authors also run a 2x2 on whether the tag lives in the pixels or the text, a permutation control for dates, and cluster permutation tests on DMV-Bench with the chain as the unit. Null results are reported clearly, e.g., D2 and R1 are both null on MemLens, and D2 and D3 hurt knowledge-update questions.

  3. The method is training-free and easy to deploy in practice.

Critical weaknesses
  1. The two terms of the decomposition start from asymmetric points. ΔR starts from a reasonable retriever, whereas ΔD starts from delivering no image at all. Practical text-only memory systems, however, usually caption images at write time (e.g., MIRIX and MemVerse, both cited by the authors; four of the MemLens baselines also receive BLIP-2 captions). ΔD therefore appears to measure "pixels vs. text" rather than "pixels vs. current practice," which naturally makes it look larger. Moreover, almost all of ΔD comes from information-extraction questions (+40.98), while the other question types are close to zero. That removing the image breaks questions whose answer is in the image is not surprising in itself. A caption-matched control, with the retrieved set fixed and the images replaced by captions, would be useful to determine the size of the delivery term.

  2. The title and abstract emphasize "delivery", but the gains on the two benchmarks come from different sources. On MemLens, the gain comes almost entirely from D1: D2 and D3 together add only +1.03, while R1 seems to not contribute. On DMV-Bench, most of the gain comes from R1 on the retrieval side. On Qwen, R1's main effect is +8.83 versus +3.96 for delivery; on Gemini, R1 contributes +22.16 while delivery contributes only +0.46, which is not significant. The paper argues that delivery is more important than retrieval, yet reaches SOTA on its second benchmark mainly through a retrieval improvement, and the abstract presents the two results side by side. The authors acknowledge this difference later in the paper, but I find the overall framing confusing nonetheless.

  3. The writing is verbose, and in places the text assumes knowledge the reader does not yet have. The main text condenses a lot of experimental numbers and statistics but lacks intuitive explanations of the method design, so the reader has to keep flipping to the appendix. I suggest streamlining the main text and moving a redrawn version of Figure 3 and/or Figure 5 from the appendix into the main paper if possible, along with more clear prose.

Questions for authors
  1. For samples where the correct image is delivered but the model still answers incorrectly (Appendix D.6), does the failure occur in the model's internal localization, or in the subsequent delivery and reasoning steps?
  2. Does delivering raw pixels cost more prefill tokens than delivering captions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 84yB (Part 1/3): W1, the caption-matched control

Thank you for the careful review, and above all for the caption-matched control. Holding the retrieved set fixed and changing only what sits in each image's place sizes the delivery term directly, and we ran it that way on five answering models with a second judge (revised Section 4, Table 2; Appendix D.7). Part 3 summarizes the revision.

How much of the delivery term survives a caption depends on what the caption carries.

Against the BLIP-2 captions that MemLens describes for four of its baselines, most of it survives: on Qwen3-VL-8B with the padded gold set (R*, defined below), the pixels add 14.45 points over no image in that run (13.87 in Table 1's run), and the caption recovers 2.31 of them (Appendix D.7).

Against the answering model's own caption, as MIRIX writes it, the pixels lead by +4.34 with Table 1's retriever (p=0.003, four models) and are level at our deployed configuration.

Against a caption that a 235B model writes for every stored image, the pixels are level (+2.31, p=0.18, gold messages alone, five models; +1.27 with R*, table below) without its write-time calls. Over that caption, perfect retrieval adds +9.36, which is 7.05 points more than the pixels' +2.31 (p=0.010).

Setup (MemLens 32K, 173 answerable questions). Retrieved messages, answering model and prompt are fixed; only what sits in each image's place changes. Captions are written once per stored image without the question, by Qwen3-VL-235B-A22B or by the answering model with the same prompt. R* is Table 1's gold set, padded to ten messages ("with the gold messages alone" drops the padding; Table 16); R is Table 1's text-channel retriever; "deployed" is the configuration of Table 3. Pooled values average the models per question and test over questions (sign-flip permutation); the Pooled row gives each difference with its p in parentheses. The gold-only figures quoted above (+2.31, +9.36) are from Table 16; the table below uses R*.

Answering model235B caption − none (R*)Pixels − 235B caption (R*)Pixels − 235B caption (deployed)Pixels − own caption (R)Pixels − own caption (deployed)ΔR (R* − R)
Qwen3-VL-8B+12.72+1.73+1.16−0.58+2.31+2.31
InternVL3.5-8B+13.87+2.31−2.31+3.47−4.05+9.83
Gemma-3-27B+15.030.00−2.89+6.94−4.62+8.09
Qwen3-VL-235B+15.03+2.89−0.58——+7.51
GLM-4.5V+20.81−0.58+2.31+7.51+2.89+1.73
Pooledall p<0.001+1.27 (0.47)−0.46 (0.85)+4.34 (0.003)*−0.87 (0.67)*+5.90 (0.007)

* four models; the 235B model's own caption is the 235B caption.

  • What the released pipeline delivers. MemLens's released pipeline puts only an [image] marker in each image's place, close to our no-image condition; against no image, delivery adds 11.21 points more than perfect retrieval does (p=0.002), pooled over five models with the gold messages alone.
  • What each form costs. The 235B caption runs a 235B-parameter model over every stored image: 4,491 calls at 32K (about 6.1M input and 0.6M output tokens), of which 9.4% are delivered for the benchmark's questions. An own caption costs the same number of calls. The pixels need no captioning call at write time; their price is answer-time tokens (Q2 in Part 2).
  • Which form, when. At the deployed configuration the pixels lead the 235B caption on information extraction, whose answer is in the image (+2.62), and trail on the other types (−2.14), p = 0.62 and 0.14. On DMV-Bench, where a caption can be checked for the cue, our full delivery leads a 7B caption by 5.8 points where the caption omits the cue and trails it by 4.1 where it names it (chain-level p = 0.057 and 0.012). Which details a later question needs is unknown when the caption is written; keeping the original, (D1), covers what a caption misses.
  • Second judge. GPT-5.4-mini agrees with the benchmark judge on 99.0% of 3,900 judgments; at the deployed configuration it gives −0.46 against the 235B caption and −1.16 against the own caption.
AuthorsOpenReview ↗

Response to Reviewer 84yB (Part 2/3)

Q1. Localization or reasoning, when the right image is delivered

About half of these failures are consistent with the model attributing the cue to the wrong image, the other half with the model failing to recognize the cue (revised Appendix D.6).

  • The audit rules out retrieval and navigation as causes: the target is retrieved in 98.4% / 100% of cases and is among the delivered items in 98% of our failures; where the reasoning trace names an item, the agent opens that item in 95–98% of cases.
  • To tell the two apart, we kept the full delivery and added under each image one line that the same 7B model wrote when it saw that image alone, without the instruction. Over 10 chains (1,888 probes) this gains +3.65 (9 of 10 chains, p=0.0098).
  • Without the description, about 45% of these failures fall on probes whose single-image description names the cue, consistent with attributing the cue to the wrong image; on those probes the description raises accuracy from 91.9 to 98.7. The other 55% fall where the model does not name the cue even when shown that image alone, consistent with failing to recognize it, and the description does not help (86.8 → 86.4). Because the instruction names the cue literally, part of the first recovery may be text matching.

Q2. Do pixels cost more prefill than captions?

At answer time, yes: at most 576 visual tokens per image at our resolution (longer side 768 px) against a median of 136 tokens per caption. The whole delivered input is a median of 2,405 tokens per question at 32K, a tenth of the full context (Appendix E.6). Captions move the cost to write time (Part 1). Where each stored image is queried many times, write-time captions amortize better. On MemLens at 32K, Table 1's retriever delivers 9.4% of the stored images for the benchmark's questions, so most captions are never read.

W2. Where the gains come from on each benchmark

The two benchmarks do reach their margins in different ways, and the revised Section 7 now says where each gain comes from, with the DMV-Bench split also in the Abstract. On MemLens it is the visual content. On DMV-Bench, whose baseline already delivers the pixels, the margin splits by backbone: on Qwen2.5-VL-7B, with the retrieved items and images fixed, the tag and the viewing-order note add +5.2 to +8.3 points at J = 5 to 15 (with our retriever, half of the margin or more at every J), while on Gemini 2.5 Flash (R1) carries most of it (Table 23, Appendix E.12). These hold each retriever's items fixed; the +3.96 you cite is delivery's main effect in the 2×2 at J = 50, where delivery with DualMem's retriever adds +1.39.

W3. Presentation

This is how the revised PDF takes up your suggestions (changes are visible in OpenReview's comparison with the submission):

  1. Section 4 gathers the caption control in one table (Table 2), with its details in Appendix D.7.
  2. In Section 5, (D2) and (D3) open with the failure each addresses, and (D1) now says what keeping the original buys over a caption.
  3. Section 7 states, per benchmark, which component carries the margin.
  4. Figure 3, one retrieved memory as DeliverMem delivers it and as each control alters it, now sits in Section 5, using the revision's additional page.
  5. Throughout the main text, long sentences are split, the padded gold set and the deployed configuration are defined at their first use in the main text, and Section 5 gives the reason behind (D2) and (R1), for instance why (R1) matches the query against each region separately.

Your review is behind three of the main changes: the caption-matched design of Table 2, the per-benchmark account of where the margin comes from (Abstract and Section 7), and moving Figure 3 into the main text. Thank you.

AuthorsOpenReview ↗

Revision summary (Part 3/3)

In short. All four reviews describe the paper as isolating the delivery stage, between retrieval and the answer, and that is what we set out to show. The revision keeps that claim and puts the evidence for it first.

  1. Delivery is a stage of its own: which parts of the retrieved memory reach the answering model, and in what form. Against no image, delivering the visual content of the retrieved messages adds 11.2 points more than perfect retrieval does with the gold messages alone, pooled over five answering models.
  2. The text-only published MemLens agents lose that content before their answering model. The benchmark's released pipeline puts an [image] marker in each image's place, a BLIP-2 caption recovers 2.3 of the 14.5 points the pixels add with the padded gold set in that run, and given the same 235B captions, Mem0, the strongest training-free agent, scores 27.75 on answerable questions and our pipeline, with no pixels at any stage, 40.46.
  3. Pixels and a caption written by a 235B model over every stored image are level at our deployed configuration; that caption takes 4,491 calls of the 235B model at 32K, while the pixels need no captioning call and cost more tokens at answer time. Neither form wins everywhere: a caption suffices when it holds what the question needs, which is unknown when it is written, and keeping the original, (D1), covers what it misses.
  4. DeliverMem is built from known parts (keep the original, give each item a readable identity, state when it was seen) and trains nothing; it leads the strongest published agent on MemLens at all four lengths and DMV-Bench's strongest method at every setting. On DMV-Bench, whose baseline already delivers the pixels, delivery with retrieval and images fixed adds 5 to 8 points at J = 5 to 15 on Qwen2.5-VL-7B.
  5. The stage leaves open questions that the decomposition makes measurable: which form to deliver for which question, where an item's identity should live (in the pixels it gains +7.89 and +8.11, in two runs, over the same tag in the text), and how a delivery cue interacts with the answer (a tag can itself be given as the answer; Q4 to bS9q, Q2 to fTEu).

Other changes. The Abstract is rewritten around these points; the Contributions put the stage first and DeliverMem as a deliberately simple instantiation; Figure 3 moves into Section 5; Section 4 adds the caption control (Table 2); "grows with backbone strength" is dropped; Section 6 adds Mem0 on our backbone; Appendix D.7 and E.12 are new, D.6 and E.8 are extended, and the prose is tightened throughout.

Thank you again for your time and careful reading. Your questions have made the paper's claim more precise, and we would be glad to clarify anything further during the discussion.

Comments on the paper

2 comments

Dear Reviewers,

The authors have submitted their rebuttal. Please review it and provide further comments as appropriate.

Best, Your AC

AuthorsOpenReview ↗

Summary of the responses and the revision

We thank the four reviewers and the AC. The details are in each thread, and this note collects the shared points.

  1. Captions (84yB W1, gkaV Q1/W1, fTEu Q1/W1, bS9q Q3). Table 2 replaces each image with a caption, on five answering models with a second judge. On Qwen3-VL-8B, a BLIP-2 caption, which MemLens describes its text-only agents as receiving, recovers only 2.31 of the 14.45 points the pixels add (padded gold set, Appendix D.7). A 235B caption performs similarly to the pixels at our deployed configuration, but it takes 4,491 calls of the 235B model at 32K when the images are stored, where the pixels need no captioning call. Against the answering model's own caption, the pixels lead by 4.34 points with our text-channel retriever (four models, p=0.003) and are level at the deployed configuration. Against withholding the images, with the gold messages alone as the ceiling, delivery adds 11.21 more points than perfect retrieval, pooled over five models (p=0.002). Which form is better depends on whether the caption holds what the question needs, which is unknown when the caption is written.

  2. Beyond restoring the images (bS9q W1/W2/Q2, gkaV W3). On DMV-Bench, whose baseline already delivers the images, the item tags and viewing-order notes add 5.22 to 8.25 points on Qwen2.5-VL-7B with 5 to 15 sessions in memory and the retrieved items fixed (Table 23). On Gemini 2.5 Flash, where 92% of DualMem's failures are retrieval misses, they add under one point and most of the margin comes from (R1), as condition (i) of Section 1 predicts. On MemLens, accurate dates add 8.76 points over no date on temporal-reasoning questions (Appendix E.7).

  3. Other model families (gkaV Q2/W2, bS9q Q5). On MemLens, with the gold messages alone as the ceiling, delivery exceeds perfect retrieval on each of the five models of Table 2, including InternVL3.5-8B, Gemma-3-27B and GLM-4.5V from outside the Qwen family. The difference is significant on three of them after Holm correction and pooled over all five (p=0.002).

  4. Published agents (gkaV W4). With the same 235B captions on our backbone, Mem0 scores 27.75 on answerable questions and our pipeline, without pixels, 40.46. At 32K the strongest published agent, MemAgent-7B, trains its memory policy by reinforcement learning, while DeliverMem trains nothing and leads every published agent at all four context lengths (Table 3).

  5. Knowledge update (bS9q Q4, fTEu Q2/W2). Over all 789 questions, the tag and the date together cost knowledge-update questions 8.62 points, mostly through answers that give a tag such as "M2" instead of the item. One instruction sentence reduces this loss to 3.45 points (n.s.) and keeps the temporal gain (Appendix E.8).

  6. Presentation (84yB W2/W3). Table 2 gathers the caption control, Figure 3 moves into Section 5, the Abstract and Section 7 state which component accounts for our lead on each benchmark, and long sentences are split.

  7. Contribution (bS9q W1, gkaV W3). The contribution is identifying delivery as a separate stage and showing when each delivery decision helps. To our knowledge, none of the related systems in Section 2 holds the retrieved set fixed and varies only what reaches the model. On the 173 answerable questions at 32K, our text-channel pipeline without images trails MemAgent-7B (27.17 against 35.26), and delivering the images, with nothing trained, takes it to 42.20 (Table 8). DeliverMem is deliberately simple, and each decision is tested against a control that alters only its own variable.

Thank you again for your careful reviews, which shaped much of this revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.