acceptodds
Under review as a conference paper at ICLR 2027

CSMem: Cognitive State Memory for Multimodal Long-Term Dialogue Assistants

Abstract

Multimodal long-term dialogue memory asks assistants to answer later questions from past image-text interactions. Existing multimodal memory methods preserve useful content and evidence through textualized records, raw multimodal memories, or graph-structured representations. However, these methods primarily preserve what appeared, rather than explicitly modeling the interaction state that gives those records meaning in context, including how references are resolved, alternatives are compared, decisions evolve, and claims are supported. To address this limitation, we introduce Cognitive State Memory (CSMem), a training-free memory framework. Inspired by situation-model accounts of comprehension, CSMem represents the structured state established within each coherent interaction rather than storing only its surface image-text records. It connects this state with retrieval cues and raw multimodal evidence to locate relevant interactions and generate grounded answers. Experiments on Mem-Gallery and LoCoMo show that CSMem outperforms textualized and raw multimodal methods, achieving up to an 11.3% relative improvement in exact match. Our code will be made publicly available soon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal long-term memory, LLM agent memory, Long-term dialogue, Retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tn1rOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CSMem, a training-free framework for multimodal long-term conversational memory. The method partitions conversation history into intra-session episodes and constructs a retrieval index, a structured state, and raw evidence for each episode. The state comprises five fields—participants, coreference bindings, comparison dimensions, state trajectories, and evidence scope—which serve to organize the context for responses and select supporting evidence.

Critical strengths

This paper addresses a specific and significant issue in multimodal memory: retrieving relevant images or text does not guarantee the correct resolution of references or the final selection. The proposed method preserves original evidence and source links; by comparing various baselines within a unified framework for both model construction and answering, it provides preliminary evidence of the system's effectiveness.

Critical weaknesses

The core contribution of this paper should lie in how specific state representations enhance memory, rather than merely adding structured fields or preprocessing steps. Existing work like A-Mem has already established precedents for converting historical content into structured records that incorporate contextual understanding—generating descriptions, keywords, and tags via LLMs to create associations and update existing memory contexts as new information arrives. Consequently, this paper needs to demonstrate the added value of its specific organizational framework—comprising participants, bindings, comparisons, trajectories, and scopes. RaMem, a study closely aligned with the motivation of the current paper, explicitly distinguishes between "content relevance" and "utility as evidence for the current query"; it anchors memories using contextual conditions such as event time, session scope, and participants, while preserving these conditions during retrieval. RaMem also conducted controlled experiments—holding memory content constant while shuffling contextual fields—to verify the efficacy of correct content-context bindings. Although the current paper cites RaMem, it fails to provide a direct comparison or similar controlled validation.

In Table 3, the conflict detection F1 score for CSMem is only 41.98%—significantly lower than OmniMem’s 82.72% and below that of the other listed external baselines. As shown in Table 2, removing Scope and Trajectory resulted in overall F1 drops of only 0.72 and 1.14 percentage points, respectively. The current results remain insufficient to support strong claims regarding the capability to model decision evolution, conflicts, and evidence boundaries.

The critical "w/o State" ablation removes both the five-field state and the additional episode-based evidence expansion. The drop in the full system's F1 score from 71.02% to 63.87% merely indicates that the combination of these two components is beneficial; it fails to demonstrate that the state representation itself is the specific source of this performance decline. While the authors have explicitly disclosed this design, such disclosure does not eliminate the confounding factor. To verify the core contribution, one should hold the retrieval results and raw evidence constant while toggling the provision of the state, and subsequently hold the state constant while independently varying the evidence expansion. The current experiments lack these two fundamental controls. The "Reference" and "Contrast" ablations simultaneously disable their corresponding evidence selection and answer generation functions, making it impossible to distinguish whether gains stem from the stored representation or from processing rules applied during the query phase.

Section 3.4 and Appendix C explain that for certain questions involving image references and dates, the system returns source-based answers directly, whereas other questions rely on task-specific formatting and post-processing. These rules are highly correlated with the EM metric. Although the paper claims that the baseline shares these rules—arguing against claims of unfair comparison—it fails to report the proportion of questions answered directly, the extent to which answers were modified by post-processing, or the results obtained when these rules are removed. The answer prompt in Appendix J still contains an unexpanded {task_policy} placeholder, and key implementation details remain undisclosed. The available information is insufficient to determine what proportion of the performance gains derives from memory versus answer processing tailored to the benchmark.

Questions for authors

see weaknesses.

14 comments
AuthorsOpenReview ↗

[part 1/5] Response to Reviewer Tn1r

Thank you for the detailed and rigorous review. We agree with the standard you set: the paper must show the value of the specific representation through controls, not only through end-to-end gains. In response, we:

  • ran the controls you specified on the full Mem-Gallery set (1,711 questions, Table 1 configuration);
  • disclose the complete answering protocol;
  • calibrate our claims.

Δ is given in percentage points relative to Full CSMem in Table 1.

W1. The added value of this specific organization (vs. A-MEM); RaMem. We agree that LLM-built structured memory is not new in itself. The question is what this particular organization adds, and we tested it directly: we fixed the retrieved episodes and the answering path, and replaced only the state.

Only the state is replacedΔF1ΔEMΔBLEU-1
(a)No state (same raw-evidence package)−2.98−4.03−3.16
(b)Budget-matched free-form summary (1.18× the state's length)−2.12−2.57−2.15
(c)A-MEM-style note (description, keywords, tags)−1.19−2.22−1.36
(d)Shuffled state (same raw-evidence package)−2.02−2.81−2.17

The same construction model generates each substitute from the same payload, and the substitute is used wherever the state is used. End to end, A-MEM trails CSMem by 4.60 F1 / 5.50 EM / 5.16 BLEU-1 (Table 1).

RaMem-style control. As in RaMem, we held the memory content fixed, meaning the same retrieved episodes and the same raw-evidence package. We then shuffled the contextual record, which here is the state, across episodes of the same scenario. This removes 70% of the state's EM benefit (2.81 of 4.03), so what matters is that content and state are correctly bound.

Relation to RaMem. We share RaMem's premise that relevance is not the same as usefulness as evidence, and our Sec. 1 makes a closely related point.

  • RaMem anchors each memory to its external episodic conditions (event time, mention time, session span, participants) to judge whether a retrieved memory is valid evidence for the query.
  • CSMem records the relations an interaction establishes internally: which image a mention denotes, how candidates differ, which role a candidate ends in, and what the episode supports. It uses these relations to select and interpret evidence after retrieval.

The two approaches are complementary, and RaMem is text-only. We will discuss it in Related Work.

AuthorsOpenReview ↗

[part 2/5] Response to Reviewer Tn1r (continued)

W2. Conflict detection, Scope/Trajectory, and claim strength. We agree, and we will calibrate our claims.

Conflict detection. CSMem is not competitive on CD (41.98 vs. 82.72), and we will say so. We audited the raw outputs for all 81 CD questions (52 true conflicts, 29 non-conflicts), and the audit locates the gap at the answer-time decision:

CSMemOmniMem
Conflicts detected (recall)11 / 52 (21.2%)46 / 52 (88.5%)
Missed conflicts416
False alarms (of 29)58

Every CD error has at least one annotated clue turn in the answer context, yet 41 of the 46 errors answer "No" to a true conflict. Our CD policy answers "No" whenever "the covered evidence does not support the alleged conflict", so an incomplete fact check ends in "No". OmniMem's released CD prompt, by contrast, counts "even small factual differences" as conflicts. Removing Contrast (together with its use) still costs 14.8 CD points.

Trajectory and Scope. Their effects are category-specific:

Field removed (Table 2)ΔF1ΔEMΔBLEU-1Main category effect (ΔF1, Fig. 3d)
Reference−1.40−1.35−1.35VR −10.8 (VR = 10.2% of questions)
Contrast−0.67−0.82−0.44CD −14.8 (4.7%)
Trajectory−1.14−1.00−1.16KR −5.4 (4.7%)
Scope−0.72−0.65−0.61diffuse (≤2.2 in any category)

KR and CD are each only 4.7% of the questions, so large category effects become small aggregate ones. For example, 14.8 × 81/1,711 = 0.70, which is Contrast's entire aggregate effect. Scope is the weakest and most diffuse field (Sec. 4.3).

Claims. We will revise the abstract, Sec. 1, and Sec. 4 so that they:

  • state that the Episode State records reference bindings, contrasts, role trajectories, and evidence scope;
  • state that its measured benefits are category-specific (Reference → VR, Contrast → CD, Trajectory → KR) and modest for Scope;
  • do not claim a general capability to model decision evolution, conflicts, or evidence boundaries.
AuthorsOpenReview ↗

[part 3/5] Response to Reviewer Tn1r (continued)

W3. The two fundamental controls. We agree that disclosure is not a control, so we ran the controls.

Control 1: retrieval and raw evidence held constant, state toggled. As shown in variant (a) under W1, with the retrieved episodes and raw-evidence package frozen, withholding the state costs 2.98 F1 / 4.03 EM / 3.16 BLEU-1. This is conservative, because the frozen package was partly selected with the state's help.

Control 2: state held constant, evidence varied. We kept the state fixed and removed only the episode-level expansion, as requested:

State fixed; evidence variedΔF1ΔEMΔBLEU-1
(f) Episode-level expansion removed−0.68−1.52−0.97

The expansion on its own contributes 0.68 F1 / 1.52 EM, so it cannot explain the architecture-level drop.

Reading the architecture-level row. This row answers only from direct cue evidence, so it removes the state from the answer context, the state's role in evidence selection, and episode-level expansion at once. Variant (a) isolates a 2.98-point F1 / 4.03-point EM decline from withholding state under fixed evidence, and variant (f) isolates the 0.68-point F1 / 1.52-point EM effect of expansion. The remaining architecture-level difference cannot be assigned to either component by these controls alone. We will report both controls and rename that row "w/o State interface (direct-cue only)".


Reviewer_CLP9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose CSMem, a training-free framework for multimodal long-term dialogue memory. It organizes past interactions into episodes, with each episode keeping an index, a structured state, and the raw evidence. The state records participants, reference bindings, contrast dimensions, state trajectories, and evidence scope. The index handles retrieving relevant episodes, while the state guides evidence selection and answer generation. Experiments on Mem-Gallery and LoCoMo report gains over nine reproduced baselines: 71.02% F1 and 54.30% EM on Mem-Gallery, and 56.02% F1 on LoCoMo. The paper also reports ablations, a retrieval analysis, and efficiency measurements.

Critical strengths

1.The paper targets a genuine issue in conversational memory: retrieving past content alone does not always capture references, comparisons, and how decisions evolved. Episode State gives this information a structured home while keeping the original evidence intact, which I find a reasonable design.

2.CSMem beats nine reproduced baselines on the main aggregate metrics across both benchmarks — a 4.60-point F1 and 5.50-point EM gain on Mem-Gallery, and a 5.25-point F1 gain on LoCoMo. The ablations and category-level breakdown are also worth reading; the effect of Contrast on conflict detection and Reference on visual reasoning are particularly interesting.

Critical weaknesses

1.Prior work, e.g. REMem (ICLR 2026), has also looked at structured episodic memory with contextual information. I would like to see a clearer account of what the five-field representation buys over these existing methods. A comparison against a plain episode representation would also go a long way toward establishing the practical value of this design.

2.In Section 4.3, "w/o State" removes the state representation and the episode-level evidence expansion at the same time. So the F1 drop from 71.02% to 63.87% cannot be read as the effect of Episode State alone — part of it may simply come from losing the expanded evidence rather than the state itself. The per-field ablations show sensible category-specific effects, but they do not establish whether the full five-field representation is better than a simpler alternative. A more controlled ablation that separates the effects of Episode State and evidence expansion would make this much more convincing.

Questions for authors
  1. How does CSMem differ from existing episodic memory methods such as REMem, and what are the main advantages of the five-field representation?
  2. Can you disentangle the effects of Episode State and episode-level evidence expansion? For instance, variants that remove each component independently under a comparable context budget would clarify their respective contributions. A comparison with a simpler episode summary would also help.
  3. In Table 3, CSMem gets 41.98% F1 on Conflict Detection versus 82.72% for OmniMem. Given that Contrast helps this category in the ablation, what accounts for the remaining gap?
6 comments
AuthorsOpenReview ↗

[part 1/3] Response to Reviewer CLP9

Thank you for the careful review and for highlighting the Contrast–CD and Reference–VR effects. In response, we:

  • reproduced REMem;
  • added a plain episode-summary baseline;
  • separated the state from the evidence pipeline;
  • audited all CD questions.

All runs use the full Mem-Gallery set (1,711 questions) with the Table 1 configuration. Δ is given in percentage points relative to Full CSMem in Table 1.

W1 / Q1. How CSMem differs from REMem, and what the five fields buy. We will cite and discuss REMem. REMem turns each session into time-stamped event gists, which are concise sentences about participants, actions, objects, and locations, together with time-scoped (subject, predicate, object) facts. It links these in a hybrid graph and answers with an agentic retriever that iterates over the graph. Both works draw on situation models, but they make different things explicit:

Made explicitLink to sourceModalityUsed for
A-MEMper-note description, keywords, tags; links between notesnote keeps original contenttextlinking, evolving and retrieving notes
REMemevent gists + time-scoped factstied to their sessiontextmulti-step graph retrieval
CSMemrelations inside an interaction: mention→participant/image bindings, contrast axes, role trajectories, evidence scopeevery record keeps source-turn and image IDs; raw turns and images retainedimage + textselecting and interpreting raw evidence after retrieval

A gist records what happened in an event. It is not designed to record which image a later "the third one" denotes, along which axes candidates were compared, or how a candidate's role changed within the episode. REMem is also text-only. CSMem stores these relations as typed records that point to the exact turns and images.

Full REMem reproduction. We ran REMem on all 20 scenarios and 1,711 questions with GPT-4o-mini for construction and Qwen3-VL-8B-Instruct for answering, the same models as for all methods. We kept its gist extraction, hybrid graph, and multi-step agentic retrieval (at most 3 steps, qa_top_k = 10) and used the official Mem-Gallery answer protocol.

Baselines relative to CSMem.

MethodΔF1ΔEMΔBLEU-1
REMem (ICLR 2026)−6.32−7.54−6.67
A-MEM (strongest baseline in Table 1)−4.60−5.50−5.16

Plain episode representation, as you suggested. We kept CSMem's own pipeline and the same retrieved episodes, and replaced only the state. A budget-matched free-form summary, the closest in-pipeline analogue of an event gist, costs 2.12 F1 / 2.57 EM / 2.15 BLEU-1 relative to the full state.

Main advantages of the five fields. The numbers are the ΔF1 when each field is removed (Fig. 3d):

  • Bindings resolve indirect mentions to images (VR −10.8).
  • Contrasts keep similar candidates distinguishable (CD −14.8).
  • The trajectory separates intermediate choices from final ones (KR −5.4).
  • Scope has the weakest and most diffuse effect (Sec. 4.3).
  • Participants anchor all fields to shared IDs.
AuthorsOpenReview ↗

[part 2/3] Response to Reviewer CLP9 (continued)

W2 / Q2. Separating Episode State from evidence expansion. We agree that 71.02 → 63.87 is not the effect of the state alone. As Sec. 4.3 and App. H state, that architecture-level row answers only from the direct cue evidence and its images, so it removes the state, the expansion, and all state-consuming steps together. We ran the separate controls below; for the controlled variants, everything not listed is identical to Full CSMem.

VariantΔF1ΔEMΔBLEU-1
(a)State withheld; same retrieved episodes and raw-evidence package−2.98−4.03−3.16
(b)Budget-matched free-form episode summary in place of the state−2.12−2.57−2.15
(f)State kept; episode-level evidence expansion removed−0.68−1.52−0.97
Table 2 "w/o State" (architecture-level; direct-cue evidence only)−7.15−5.03−7.00

In (b), the same construction model generates the substitute from the same episode payload, and the summary replaces the state everywhere the state is used. It is on average 1.18× the state's length.

  • State alone (a). With the same retrieved episodes and raw-evidence package, withholding the state costs 2.98 F1 / 4.03 EM / 3.16 BLEU-1. This is conservative, because the frozen package was partly selected with the state's help.
  • Plain representation (b). The budget-matched episode summary also falls short of the five-field state.
  • Evidence varied with the state fixed (f). Removing episode-level expansion costs 0.68 F1 / 1.52 EM / 0.97 BLEU-1.

Variant (a) accounts for 2.98 of the 7.15 F1 points and 4.03 of the 5.03 EM points (69 of 86 questions); variant (f) shows that expansion alone accounts for 0.68 F1 points. The remaining architecture-level difference also includes the state's role in evidence selection, so it should not be attributed to the state representation alone. We will add these controls to Sec. 4.3 and rename the Table 2 row "w/o State interface (direct-cue only)".

AuthorsOpenReview ↗

[part 3/3] Response to Reviewer CLP9 (continued)

Q3. The CD gap to OmniMem. We audited the raw model outputs for all 81 CD questions: 52 contain a true conflict and 29 do not.

CSMemOmniMem
Conflicts detected (recall)11 / 52 (21.2%)46 / 52 (88.5%)
Missed conflicts416
False alarms (of 29)58
Correct35 / 8167 / 81

After post-processing, CSMem's reported CD score is 34/81 = 41.98% (Table 3). The remaining gap is mainly an answer-time decision gap rather than a retrieval failure:

  • Every CD error has at least one annotated clue turn in the answer context.
  • 41 of the 46 errors (89%) answer "No" to a true conflict. For example, the memory says Lumi is a dog and the statement says a cat, yet the model still answers "No". In the other direction, we raise fewer false alarms than OmniMem (5 vs. 8).
  • Our CD task_policy answers "No" both when the claim is consistent with memory and when "the covered evidence does not support the alleged conflict". An incomplete fact check therefore ends in "No", and CD post-processing leans the same way (35 → 34 correct). OmniMem's released CD prompt instead requires a fact-by-fact comparison and counts "even small factual differences" as conflicts.

On why Contrast helps but does not close the gap: removing Contrast (together with its use) costs 14.8 CD points, about 12 of the 81 questions, because Contrast records which attributes distinguish entities. CD additionally asks whether two statements concern the same entity, event, and time, and whether both can hold. Recording a difference is not the same as judging a contradiction. We will state explicitly that CSMem is not competitive on CD and report this audit. All reported results keep the original policy.

We hope the REMem comparison, the plain-summary baseline, and the separation of state from expansion address your concerns.

Reviewer_WjzfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CSMem, a training-free memory framework for multimodal long-term dialogue. It organizes each episode into a retrieval index, an explicit interaction state, and linked raw evidence. The state represents participants, reference bindings, contrast dimensions, state trajectories, and evidence scope. Experiments on Mem-Gallery and LoCoMo show improvements over several reproduced memory baselines, supported by component ablations, retrieval diagnostics, and efficiency analyses.

The paper addresses a meaningful problem and presents a coherent system. However, the current experiments establish the effectiveness of the combined pipeline more clearly than the specific value of its proposed state representation.

Critical strengths
  1. Well-motivated representation. Distinguishing remembered content from its role in an interaction is useful. Explicitly preserving references, candidate comparisons, and evolving preferences offers a plausible way to reduce repeated reconstruction at query time. Linking these records to source evidence is also a sensible design choice.

  2. Promising aggregate results under shared backbones. CSMem achieves 71.02% F1 and 54.30% EM on Mem-Gallery, compared with 66.42% and 48.80% for A-MEM, and improves LoCoMo overall F1 from M2A’s 50.77% to 56.02%. Using common construction and answering models reduces an important source of comparison variability.

  3. Useful diagnostic coverage and transparent limitations. The paper reports category-level results, field ablations, retrieval coverage, hyperparameter sensitivity, and query latency. Selective exposure improves accuracy while reducing average input tokens from 6,392 to 3,336. The authors also explicitly acknowledge the confounded state ablation and missing matched-capacity comparison.

Critical weaknesses

W1. The central representational contribution is not sufficiently isolated.

The “w/o State” variant removes both the state records and additional episode-level evidence expansion (Section 4.3). Consequently, its 7.15-point F1 drop cannot distinguish better interaction-state modeling from improved evidence availability. The paper appropriately acknowledges this, but the distinction remains central to evaluating the contribution. Moreover, there is no comparison against a free-form episode summary or generic structured memory with comparable construction and context budgets. Such controls would help determine whether the five-field organization contributes beyond additional preprocessing and richer episode context.

W2. Important parts of the answering protocol remain underspecified.

Section 3.4 and Appendix C allow source-based rules to answer exact image-reference and date questions directly, followed by task-specific formatting and postprocessing for other cases. These operations may materially affect the reported scores, particularly visual-search performance. Although the paper states that the rules are shared across methods and category labels are withheld, it does not sufficiently specify the routing criteria, direct-answer algorithms, or the {task_policy} in Appendix J. This limits reproducibility and makes it difficult to separate memory quality from specialized answer handling. This is a protocol-transparency concern, not evidence of annotation leakage.

W3. The extracted states are not directly validated.

The paper evaluates downstream answers but does not measure whether extracted bindings, trajectories, and scope judgments accurately represent the source dialogue. This matters because these records guide both evidence selection and answering. The relatively small aggregate effects of removing trajectory or scope also leave their contribution uncertain. Furthermore, Table 3 shows weak conflict detection—41.98% F1 versus OmniMem’s 82.72%—despite the emphasis on evidence support. A targeted audit could clarify whether these failures arise from extraction, retrieval, or downstream use of otherwise correct states. The heuristic answer-error taxonomy in Appendix F does not resolve this question.

W4. Robustness and total resource requirements are incompletely characterized.

The main results use single runs and one construction/answer-model pairing. Uncertainty estimates are particularly relevant for the smaller field-ablation differences. The efficiency analysis reports query latency but omits construction cost and storage, despite five field-specific extraction calls per episode. Thus, the practical trade-off between preprocessing expense and downstream gains remains unclear.

Questions for authors
  1. Can you isolate state removal while holding retrieved evidence fixed, and compare against a budget-matched episode summary or generic structured representation? Note that I do not expect a full-scale ablation study in rebuttal; a controlled comparison on a representative subset, or a convincing argument supported by existing evidence explaining why further ablation is unnecessary to address W1, would be sufficient.
  2. Can you provide the complete routing, direct-answer, and postprocessing rules, report how often direct answering is used, and evaluate performance without those shortcuts?
  3. Can you audit extracted state correctness on a representative subset and relate errors to downstream failures, especially conflict detection? A full-dataset human evaluation is not necessary.
  4. Can you provide uncertainty estimates for the main gains and key ablations, together with construction-token/API costs and storage requirements?
8 comments
AuthorsOpenReview ↗

[part 1/4] Response to Reviewer Wjzf

Thank you for the careful review and for scoping your requests so precisely. Your note that a controlled comparison would suffice for W1 directly shaped the study below. We ran the targeted comparisons on the full Mem-Gallery set (all 1,711 questions), using the construction model, answer model, and split of Table 1. Δ is given in percentage points relative to Full CSMem in Table 1.

W1 / Q1. Isolating the representational contribution. We agree that Table 2's "w/o State" cannot isolate the state. As Sec. 4.3 and App. H state, it is an architecture-level variant that evaluates the combined state-and-expansion interface. Following Q1, we froze the retrieved episodes and the raw-evidence package and toggled only the state. We also replaced the state with a budget-matched episode summary and an A-MEM-style structured note.

Controlled state study. For variants (a)-(c), everything not listed is identical to Full CSMem.

VariantΔF1ΔEMΔBLEU-1
(a)State withheld; same retrieved episodes and raw-evidence package−2.98−4.03−3.16
(b)Budget-matched free-form episode summary in place of the state−2.12−2.57−2.15
(c)A-MEM-style note (description, keywords, tags) in place of the state−1.19−2.22−1.36
Table 2 "w/o State" (architecture-level; direct-cue evidence only)−7.15−5.03−7.00

In (b) and (c), the same construction model generates the substitute from the same episode payload, and the substitute replaces the state everywhere the state is used (answer context and evidence selection). The summary is length-matched to the state: on average it is 1.18× the state's length, so it gets slightly more budget.

What the controls show:

  1. The state adds value beyond the evidence. With identical evidence, withholding the state costs 2.98 F1 / 4.03 EM (69 questions). This estimate is conservative. The frozen package was partly selected with the state's help, so (a) measures only the state's value as answer context.
  2. The organization matters, not just extra preprocessing. Both the free-form summary (despite its slightly larger budget) and the A-MEM-style structured note fall short of the five-field state.

On the 7.15-point figure. That architecture-level row removes the state from the answer context, the state's role in selecting evidence, and episode-level expansion at once, so its drop cannot be attributed to the state alone. Variant (a) isolates the state-as-answer-context effect while holding the retrieved episodes and raw-evidence package fixed. We will add this controlled comparison to Sec. 4.3 and rename the Table 2 row "w/o State interface (direct-cue only)".

AuthorsOpenReview ↗

[part 2/4] Response to Reviewer Wjzf (continued)

W2 / Q2. Answering protocol. We appreciate your framing of this as a transparency question. The complete protocol follows; App. C will give it as pseudocode.

(i) Routing and direct answers. There are two direct-read paths, and each returns an answer only when its evidence condition holds.

  • Date read (26 questions). This path applies when all of the following hold:

    • the question is of type TR and is not a before/after or yes/no temporal question;
    • the question contains a YYYY-MM-DD date or "which/what/on which date", or it starts with "when did/when was";
    • a direct cue carries a YYYY-MM-DD date and shares at least 3 content words with the question (at least 2 if the question has 6 or fewer content words).

    It returns that date.

  • Exact-image count read (46 questions). This path applies when the question is of type VR, is a counting question ("how many", "how much", "number of", "count(s)/counting"), and retrieval yields exact image matches. It returns the number of matched image IDs.

(ii) Post-processing. All answers are first whitespace-normalized. Type-specific post-processing then adjusts the answer form. For CD, it applies rule-based checks on choice history, final choice, and temporal order, and falls back to "No." when a top-ranked direct cue overlaps strongly with the claim. Post-processing changed the raw output of 142 questions (8.3%); 138 of these still differ after the scorer's normalization.

(iii) How often and how much.

VariantΔF1ΔEMΔBLEU-1
Both direct-read paths and all rewrite/post-processing rules disabled+0.57+0.76+0.36
For reference: A-MEM, strongest baseline (Table 1)−4.60−5.50−5.16

The deterministic rules are not a source of CSMem's gains: with both direct-read paths and all rewriting rules switched off, performance slightly improves (13 EM questions), and the margin over A-MEM widens to 5.17 F1 / 6.26 EM. If anything, the rules cost a little accuracy.

(iv) {task_policy}. This follows the official Mem-Gallery runner (benchmark/run/run_bench.py). The shared answer prompt (Prompt 6, App. J) is followed by a short type-specific instruction for the three types that the official runner also special-cases:

TypeOfficial runner (ar/cd/vs_prompt.txt)CSMem task_policy
AR"Provide your answer based on the information in the conversation. Only if the information about the question is not present in the conversation, reply with: 'Not mentioned.'""This is an evidence-boundary question. If the exact requested detail is not explicitly stated in the retrieved evidence, answer Not mentioned. Do not convert missing evidence into No."
CD"Please check whether this information conflicts with the conversation, and reply strictly with either 'Yes.' or 'No.'""This is a claim-verification question. Treat the question as asking whether its claim contradicts the memory. First compare the claim against Direct Cue Evidence. Answer exactly Yes. if the claim is contradicted by memory. Answer exactly No. if the claim is consistent with memory or the covered evidence does not support the alleged conflict. Do not answer Not mentioned when the direct evidence covers the topic/entity."
VS"Return the image_id of the image(s). If there are multiple images, sort them in ascending order and separate them by commas." (format example omitted)"Prefer exact image ids, product names, category labels, and visual attributes that appear in Direct Cue Evidence. For count questions, count distinct matching memory image ids from the relevant evidence only, not neighboring turns or repeated captions."

(v) What "withheld" covers. The point label is never inserted into the prompt and never stored in memory. It is not used in memory construction or in the BM25/vector ranking of cue records. It acts only at query time, as a question-type switch, in four places:

  • it selects the task_policy;
  • it enables image-group selection for visual question types;
  • it gates the two direct-read paths;
  • it selects the type-specific post-processing.

These are the "benchmark-compatible operation and answer-format rules" referred to in Sec. 3.4 and App. C, and the revision will state them in full.

AuthorsOpenReview ↗

[part 3/4] Response to Reviewer Wjzf (continued)

W3 / Q3. Validating extracted states; conflict detection.

Conflict detection. We audited the raw model outputs for all 81 CD questions: 52 contain a true conflict and 29 do not.

CSMemOmniMem
Conflicts detected (recall)11 / 52 (21.2%)46 / 52 (88.5%)
Missed conflicts416
False alarms (of 29)58
Correct35 / 8167 / 81

After post-processing, CSMem's reported CD score is 34/81 = 41.98% (Table 3).

  • Retrieval is not the main bottleneck. Every CD error has at least one annotated clue turn in the answer context.
  • The error is one-directional. 41 of the 46 errors (89%) answer "No" to a true conflict. For example, the memory says Lumi is a dog and the statement says a cat, yet the model still answers "No".
  • Likely cause. Our CD task_policy (quoted under W2) answers "No" both when the claim is consistent and when "the covered evidence does not support the alleged conflict". An incomplete fact check therefore ends in "No", and CD post-processing leans the same way (35 → 34 correct). OmniMem's released CD prompt instead requires a fact-by-fact comparison and counts "even small factual differences" as conflicts. This is consistent with its far fewer misses (6 vs. 41) and its higher number of false alarms (8 vs. 5).
  • Role of Contrast. Removing Contrast (together with its query-time use) costs 14.8 CD points, because Contrast records which attributes distinguish entities. CD also requires deciding whether two statements concern the same entity, event, and time, and whether both can hold. Recording a difference is not the same as judging a contradiction.

The CD gap is therefore mainly a matter of answer-time decision calibration rather than retrieval. All reported results keep the original policy, and the revision states plainly that CSMem is not competitive on CD.

State faithfulness. As a pilot check, we audited the extracted states record by record for source faithfulness: 111 state records across 10 episodes sampled to cover both conflict-related and control scenarios, each checked against the episode's full dialogue and captions. 101 records (91%) are directly supported by their cited source; 10 (9%) are partially supported, typically a hedged or conditional statement ("might", "if") recorded more definitively than the source; and none lacks a source. Source-turn and modality attributions are correct for all 111 records. The audit also shows where extraction can improve: some before→after changes are under-recorded, and a few details remain only in evidence quotes rather than as structured content. Because raw evidence is always retained and retrieved alongside the state, such gaps remove a cue rather than erase information. The category-specific field effects below are also consistent with the intended roles of the fields.

Field ablations (Table 2) and their main category effect (Fig. 3d).

Field removed (with its use)ΔF1ΔEMΔBLEU-1Main category effect (ΔF1)
Reference−1.40−1.35−1.35VR −10.8 (VR = 10.2% of questions)
Contrast−0.67−0.82−0.44CD −14.8 (4.7%)
Trajectory−1.14−1.00−1.16KR −5.4 (4.7%)
Scope−0.72−0.65−0.61diffuse (≤2.2 in any category)

Trajectory's small aggregate effect follows from this concentration. Scope is the weakest and most diffuse field, as Sec. 4.3 states, and we will present it that way. We will include the state-faithfulness audit and clarify these field-specific limitations in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.