- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LMMR, a framework for long-term multimodal memory reasoning that combines shared sparse semantic concepts, graph-structured memory organization, and continuous latent reasoning. The method maps textual hidden states and visual patch features into a shared sparse autoencoder concept space, uses the resulting semantic activations to construct query-conditioned multimodal memory graphs, and injects graph representations into the language model through a latent reasoning rollout. Native visual tokens are retained to preserve fine-grained visual evidence. The framework is evaluated on Mem-Gallery and MemEye, with experiments indicating improvements over several text-based and multimodal memory baselines. Ablation studies examine the contributions of textual memory, visual memory, and latent reasoning.
Critical strengths
-
The paper targets an important problem beyond conventional multimodal retrieval. Correctly retrieving historical visual content is not sufficient when answering a query requires entity resolution, cross-session integration, and temporal validity. LMMR explicitly models these relationships through a query-conditioned memory structure rather than treating retrieved images and text as independent context. This is a meaningful direction for long-term multimodal agents.
-
The proposed architecture integrates semantic organization with fine-grained visual evidence. Shared sparse concepts provide a potential mechanism for linking text and image content, while native visual tokens preserve perceptual information that would otherwise be lost through abstraction. The evaluation on both Mem-Gallery and MemEye, including transfer to an additional benchmark, is a positive aspect of the empirical study.
Critical weaknesses
-
The necessity and novelty of the shared SAE concept space are not sufficiently isolated. Existing research has already investigated multimodal sparse autoencoders and shared cross-modal concept dictionaries. LMMR introduces a particular application of such representations to graph-based memory reasoning, but the current evidence does not establish why sparse concept addresses are essential for the observed improvements. Comparisons with dense cross-modal embeddings, conventional contrastive alignment, or alternative concept-linking mechanisms under the same graph architecture would be important for substantiating the contribution.
-
The current ablations do not fully separate the effect of latent reasoning from the effect of structured evidence integration and model capacity. Removing the latent reasoning component produces a large accuracy decrease, but this does not by itself demonstrate that a continuous latent rollout is superior to other mechanisms for incorporating graph information. A parameter-matched alternative that directly conditions the decoder on the same graph representation would provide a more decisive comparison. Moreover, the additional complexity of SAE construction, graph encoding, and latent rollout deserves a clearer end-to-end efficiency analysis.
Questions for authors
-
Can the authors replace the shared SAE dictionary with a dense shared representation trained using standard cross-modal alignment, while keeping the same graph construction, reasoning backbone, and training budget? How much of the performance gain specifically depends on sparse concept addressing?
-
Can the authors compare latent graph injection against direct graph-to-decoder conditioning or cross-attention fusion using the same retrieved evidence and comparable parameter counts? This would clarify whether the benefit comes from the continuous latent reasoning process or from additional structured evidence integration.
-
How is temporal validity operationalized in the memory graph? In particular, how does the method distinguish newer observations that supersede earlier states from observations that merely refer to the same entity at different times? Can the authors provide targeted temporal-update and conflicting-evidence evaluations?
-
What are the end-to-end latency, peak memory usage, graph construction cost, and retrieval cost as the number of historical turns increases? How do these quantities compare with strong multimodal retrieval baselines at matched accuracy?
8 comments
Response to Reviewer Ju19 (2/5)
Answer to Weakness #1 & Question #1, continued.
This comment led us to compare the shared space with alternative spaces and linking mechanisms under the same graph, which has made its role in LMMR considerably clearer, and we are grateful for it. We have added the shared-space and linking comparisons to Sec. 5.3, Table 2, and discussed related multimodal SAE work in Sec. 2.
Weakness #2 & Question #2
The current ablations do not fully separate the effect of latent reasoning from the effect of structured evidence integration and model capacity. Removing the latent reasoning component produces a large accuracy decrease, but this does not by itself demonstrate that a continuous latent rollout is superior to other mechanisms for incorporating graph information. A parameter-matched alternative that directly conditions the decoder on the same graph representation would provide a more decisive comparison. Moreover, the additional complexity of SAE construction, graph encoding, and latent rollout deserves a clearer end-to-end efficiency analysis.
Can the authors compare latent graph injection against direct graph-to-decoder conditioning or cross-attention fusion using the same retrieved evidence and comparable parameter counts? This would clarify whether the benefit comes from the continuous latent reasoning process or from additional structured evidence integration.
Answer to Weakness #2 & Question #2.
We appreciate this precise formulation. Table 2 removes the latent reasoning from the trained model, so it measures how much the trained model relies on it. It does not by itself separate latent reasoning from other ways of passing the graph output to the decoder.
To make this separation, we compare three ways of passing the graph output to the decoder, with the same retrieved evidence, graph, backbone and training recipe. LMMR passes it through latent reasoning. w/ Direct Conditioning maps it to a soft input token and decodes the answer directly. w/ Cross-Attention lets a few learned queries attend over the graph's node states to produce the input tokens.
Table R2a. Alternative ways of passing the graph output to the decoder: accuracy.
| Method | F1 ↑ | BLEU-1 ↑ | EM ↑ | L-Judge ↑ |
|---|---|---|---|---|
| LMMR (Full) | 79.10 | 74.79 | 63.47 | 87.35 |
| w/ Direct Conditioning | 76.51 (↓2.59) | 72.03 (↓2.76) | 60.43 (↓3.04) | 83.99 (↓3.36) |
| w/ Cross-Attention | 76.62 (↓2.48) | 72.13 (↓2.66) | 60.07 (↓3.40) | 83.99 (↓3.36) |
Passing the graph output to the decoder by direct conditioning or by cross-attention lowers the score relative to LMMR, and both remain far above MuRAG. We attribute this to the latent reasoning working on the graph representation: it integrates the graph readout with the dated turns step by step before the answer is written, whereas a single soft token or a one-shot fusion hands the graph evidence over once. In all three configurations, the structured evidence built on the shared space and the graph is what the baseline lacks, and the latent reasoning makes the fullest use of it.
End-to-end cost. The three configurations were also timed on one GPU, and Table R2b reports the time of each stage. Selection, the SAE with the graph, prefill and decoding are the same in all three, and the SAE and the graph together take under 7 ms, so the only stage that differs is the interface to the decoder. Their interfaces have a matched number of trainable parameters, so the comparison is not one of model capacity, and all three answer within the same order of time, about one to two seconds per query, with the same memory on top of the backbone weights. The comparison with the baselines and the scaling with the history length are in our response to Q4 (Tables R6 and R7).
Table R2b. Alternative ways of passing the graph output to the decoder: per-query time by stage.
| Metric | LMMR (Full) | w/ Direct Conditioning | w/ Cross-Attention |
|---|---|---|---|
| F1 ↑ | 79.10 | 76.51 | 76.62 |
| BLEU-1 ↑ | 74.79 | 72.03 | 72.13 |
| EM ↑ | 63.47 | 60.43 | 60.07 |
| L-Judge ↑ | 87.35 | 83.99 | 83.99 |
| Candidate selection and input assembly (ms) ↓ | 19.5 | 19.0 | 18.1 |
| SAE and graph (ms) ↓ | 6.7 | 6.7 | 6.7 |
| Prefill of the multimodal prefix (ms) ↓ | 815 | 809 | 813 |
| Interface to the decoder (ms) ↓ | 496 | 90 | 97 |
| Answer decoding (ms) ↓ | 442 | 462 | 445 |
| Total latency (ms) ↓ | 1,781 | 1,388 | 1,381 |
| Trainable parameters of the interface (M) | 12.9 | 12.9 | 9.4 |
| Peak GPU memory over the weights (GB) ↓ | 1.9 | 1.9 | 1.9 |
Response to Reviewer Ju19 (3/5)
Answer to Weakness #2 & Question #2, continued.
LMMR reaches the higher scores of Tables R1 and R2a at a similar cost, and the extra time it spends, below half a second per query, is the interface itself, latent reasoning, which integrates the graph readout with the dated turns step by step before the answer is written. This is a trade-off between accuracy and time, and we consider it acceptable in the setting that LMMR is built for: a long-term assistant that answers once per user turn, where a response in under two seconds is natural in a conversation and where answering correctly about an earlier event, an update or a conflict is worth far more than a few tenths of a second.
What the existing results also separate. The gain is not more evidence: LMMR reads fewer retrieved tokens than the larger budgets of MuRAG, whose accuracy stays flat as its budget grows, and enlarging LMMR's own visible-image budget does not help (Table 3, Fig. 8). It is not more computation either, since the effect of latent reasoning saturates after a couple of steps (Fig. 8). All systems use the same backbone, and a latent step re-uses its weights (Eq. 6). The cost of the SAE, the graph and latent reasoning is broken down by stage in our response to Q4.
Your comment helped us state more clearly what Table 2 measures, and we are grateful for it. We have added the accuracy controls to Sec. 5.3, Table 2, the interface timings to Appendix C.3, Table 15, and the full-system stage breakdown to Appendix A.4, Table 10. Sec. 5.4 summarizes the runtime results.
Question #3
How is temporal validity operationalized in the memory graph? In particular, how does the method distinguish newer observations that supersede earlier states from observations that merely refer to the same entity at different times? Can the authors provide targeted temporal-update and conflicting-evidence evaluations?
Answer to Question #3.
We appreciate this question on temporal validity and reference, which is central to long-term memory. Deciding which state is valid takes two things: recognising that a later statement or image is about the same entity as an earlier one, and knowing which of the two is later. LMMR uses neither a hand-written recency rule nor an entity label for this. It gives the backbone dated, ordered evidence in which every image stays in the turn it came from, and it learns, from questions where the distinction decides the answer, whether a later statement replaces an earlier one or only mentions the same entity.
How validity and reference are handled.
- Provenance (Sec. 4.2). Every textual turn and every image entry stores its dialogue position and date, and the selected turns are shown in chronological order together with them.
- No name–image binding (Sec. 4.3). LMMR never ties a name to an image or an entity to a state. An image stays in its source turn, and the text, caption and image of that turn form one node of the graph. A later mention such as "the tag" is therefore read by the backbone together with the turns around it. There is no separate coreference module, and an image with two people is treated like any other image.
- Temporal message (Sec. 4.3). A turn node receives its temporal message from the current and earlier turns only, with attention weights , so a later turn is read relative to the observation it may revise. The concept message connects turns that activate the same dictionary entries. We have not ablated the temporal message on its own, so the evaluations below do not separate its effect from the rest of the model.
- Supervision. The answer loss covers the question types that depend on this distinction, and evaluation uses held-out conversations.
Targeted evaluations.
(a) Update, conflict and time questions in Mem-Gallery. Knowledge resolution asks for the currently valid fact after an update, conflict detection whether the history contradicts itself, and temporal reasoning for the order and dates of events across sessions. Table R3 gives LMMR minus the best baseline (Table 9). Conflict detection is a binary decision, which is why its four columns agree.
Response to Reviewer Ju19 (1/5)
We sincerely appreciate your careful reading and your positive assessment of our work. We hope the following responses fully resolve the concerns you raised, and we would be grateful for your continued support as we finalize the paper.
Answer to Weakness #1 & Question #1.
We appreciate this thoughtful comment. Multimodal sparse autoencoders and shared concept dictionaries have indeed been studied, and we have cited and discussed this line of work in Sec. 2. Our use is different: the sparse dictionary is trained jointly with the answer generator and serves as the common address space through which the text and the images of a long dialogue meet, both in the alignment objective and in the concept message of the turn graph. Unless stated otherwise, all results in this response are on Mem-Gallery with Qwen2.5-VL-7B as the backbone.
To examine whether this shared sparse space is the right choice under the same graph architecture, we replace it by several alternatives. The memory, the turn graph, the backbone, the latent reasoning and the training recipe are kept fixed, and only the shared space or the way the nodes are linked changes. w/ Dense Embedding uses a dense cross-modal embedding model (the retriever of MuRAG) as the shared space. w/ Contrastive Alignment replaces the dictionary with a dense space aligned by a standard contrastive loss. w/ Dense Link and w/ Random Link, set apart by a dashed line in the table, keep the dictionary but link the turn nodes by dense nearest neighbours, or at random.
Table R1. Alternative shared spaces and linking mechanisms under the same graph architecture.
Every alternative is below LMMR, and all of them stay well above the strongest baseline, MuRAG, so the shared sparse space and the concept-based graph are both needed for the best result. The reasons differ by alternative. With a frozen off-the-shelf embedding or a conventional contrastive space, text and images are compared by generic similarity, and nothing makes a turn and its images activate the same concept entries, so the graph cannot use shared concept addresses to bind them. With dense or random neighbours instead of concept links, the graph connects turns by overall similarity or by chance rather than by the concepts they share, so the memory state carries less of the evidence the question needs. MuRAG has neither a shared space nor a graph and falls furthest behind. In LMMR, the dictionary provides one set of discrete concept addresses for text and images, the concept message links turns through them, and the readout is a memory state tied to the right evidence. This is the advantage of the full design, and the sparse dictionary additionally provides addresses that can be read and inspected (Fig. 7).
What the shared sparse space contributes. The alignment is learned rather than given: image and text states are unrelated before training, and afterwards paired states are far closer than negatives and in-bank retrieval clearly improves, for both backbones (App. C.1–C.2). The dictionary is fitted once and frozen, and its entries remain coherent on MemEye without refitting (Fig. 7), where LMMR is evaluated zero-shot and obtains the best multiple-choice accuracy (Fig. 3). When more topics are mixed into one memory, LMMR stays stable while full visual context degrades (Fig. 4a). Dense cross-modal retrieval is already part of the comparison, since MuRAG retrieves with a dense cross-modal embedding model, and LMMR outperforms it (Table 1).