- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CSMem, a training-free framework for multimodal long-term conversational memory. The method partitions conversation history into intra-session episodes and constructs a retrieval index, a structured state, and raw evidence for each episode. The state comprises five fields—participants, coreference bindings, comparison dimensions, state trajectories, and evidence scope—which serve to organize the context for responses and select supporting evidence.
Critical strengths
This paper addresses a specific and significant issue in multimodal memory: retrieving relevant images or text does not guarantee the correct resolution of references or the final selection. The proposed method preserves original evidence and source links; by comparing various baselines within a unified framework for both model construction and answering, it provides preliminary evidence of the system's effectiveness.
Critical weaknesses
The core contribution of this paper should lie in how specific state representations enhance memory, rather than merely adding structured fields or preprocessing steps. Existing work like A-Mem has already established precedents for converting historical content into structured records that incorporate contextual understanding—generating descriptions, keywords, and tags via LLMs to create associations and update existing memory contexts as new information arrives. Consequently, this paper needs to demonstrate the added value of its specific organizational framework—comprising participants, bindings, comparisons, trajectories, and scopes. RaMem, a study closely aligned with the motivation of the current paper, explicitly distinguishes between "content relevance" and "utility as evidence for the current query"; it anchors memories using contextual conditions such as event time, session scope, and participants, while preserving these conditions during retrieval. RaMem also conducted controlled experiments—holding memory content constant while shuffling contextual fields—to verify the efficacy of correct content-context bindings. Although the current paper cites RaMem, it fails to provide a direct comparison or similar controlled validation.
In Table 3, the conflict detection F1 score for CSMem is only 41.98%—significantly lower than OmniMem’s 82.72% and below that of the other listed external baselines. As shown in Table 2, removing Scope and Trajectory resulted in overall F1 drops of only 0.72 and 1.14 percentage points, respectively. The current results remain insufficient to support strong claims regarding the capability to model decision evolution, conflicts, and evidence boundaries.
The critical "w/o State" ablation removes both the five-field state and the additional episode-based evidence expansion. The drop in the full system's F1 score from 71.02% to 63.87% merely indicates that the combination of these two components is beneficial; it fails to demonstrate that the state representation itself is the specific source of this performance decline. While the authors have explicitly disclosed this design, such disclosure does not eliminate the confounding factor. To verify the core contribution, one should hold the retrieval results and raw evidence constant while toggling the provision of the state, and subsequently hold the state constant while independently varying the evidence expansion. The current experiments lack these two fundamental controls. The "Reference" and "Contrast" ablations simultaneously disable their corresponding evidence selection and answer generation functions, making it impossible to distinguish whether gains stem from the stored representation or from processing rules applied during the query phase.
Section 3.4 and Appendix C explain that for certain questions involving image references and dates, the system returns source-based answers directly, whereas other questions rely on task-specific formatting and post-processing. These rules are highly correlated with the EM metric. Although the paper claims that the baseline shares these rules—arguing against claims of unfair comparison—it fails to report the proportion of questions answered directly, the extent to which answers were modified by post-processing, or the results obtained when these rules are removed. The answer prompt in Appendix J still contains an unexpanded {task_policy} placeholder, and key implementation details remain undisclosed. The available information is insufficient to determine what proportion of the performance gains derives from memory versus answer processing tailored to the benchmark.
Questions for authors
see weaknesses.
14 comments
[part 2/5] Response to Reviewer Tn1r (continued)
W2. Conflict detection, Scope/Trajectory, and claim strength. We agree, and we will calibrate our claims.
Conflict detection. CSMem is not competitive on CD (41.98 vs. 82.72), and we will say so. We audited the raw outputs for all 81 CD questions (52 true conflicts, 29 non-conflicts), and the audit locates the gap at the answer-time decision:
| CSMem | OmniMem | |
|---|---|---|
| Conflicts detected (recall) | 11 / 52 (21.2%) | 46 / 52 (88.5%) |
| Missed conflicts | 41 | 6 |
| False alarms (of 29) | 5 | 8 |
Every CD error has at least one annotated clue turn in the answer context, yet 41 of the 46 errors answer "No" to a true conflict. Our CD policy answers "No" whenever "the covered evidence does not support the alleged conflict", so an incomplete fact check ends in "No". OmniMem's released CD prompt, by contrast, counts "even small factual differences" as conflicts. Removing Contrast (together with its use) still costs 14.8 CD points.
Trajectory and Scope. Their effects are category-specific:
| Field removed (Table 2) | ΔF1 | ΔEM | ΔBLEU-1 | Main category effect (ΔF1, Fig. 3d) |
|---|---|---|---|---|
| Reference | −1.40 | −1.35 | −1.35 | VR −10.8 (VR = 10.2% of questions) |
| Contrast | −0.67 | −0.82 | −0.44 | CD −14.8 (4.7%) |
| Trajectory | −1.14 | −1.00 | −1.16 | KR −5.4 (4.7%) |
| Scope | −0.72 | −0.65 | −0.61 | diffuse (≤2.2 in any category) |
KR and CD are each only 4.7% of the questions, so large category effects become small aggregate ones. For example, 14.8 × 81/1,711 = 0.70, which is Contrast's entire aggregate effect. Scope is the weakest and most diffuse field (Sec. 4.3).
Claims. We will revise the abstract, Sec. 1, and Sec. 4 so that they:
- state that the Episode State records reference bindings, contrasts, role trajectories, and evidence scope;
- state that its measured benefits are category-specific (Reference → VR, Contrast → CD, Trajectory → KR) and modest for Scope;
- do not claim a general capability to model decision evolution, conflicts, or evidence boundaries.
[part 3/5] Response to Reviewer Tn1r (continued)
W3. The two fundamental controls. We agree that disclosure is not a control, so we ran the controls.
Control 1: retrieval and raw evidence held constant, state toggled. As shown in variant (a) under W1, with the retrieved episodes and raw-evidence package frozen, withholding the state costs 2.98 F1 / 4.03 EM / 3.16 BLEU-1. This is conservative, because the frozen package was partly selected with the state's help.
Control 2: state held constant, evidence varied. We kept the state fixed and removed only the episode-level expansion, as requested:
| State fixed; evidence varied | ΔF1 | ΔEM | ΔBLEU-1 |
|---|---|---|---|
| (f) Episode-level expansion removed | −0.68 | −1.52 | −0.97 |
The expansion on its own contributes 0.68 F1 / 1.52 EM, so it cannot explain the architecture-level drop.
Reading the architecture-level row. This row answers only from direct cue evidence, so it removes the state from the answer context, the state's role in evidence selection, and episode-level expansion at once. Variant (a) isolates a 2.98-point F1 / 4.03-point EM decline from withholding state under fixed evidence, and variant (f) isolates the 0.68-point F1 / 1.52-point EM effect of expansion. The remaining architecture-level difference cannot be assigned to either component by these controls alone. We will report both controls and rename that row "w/o State interface (direct-cue only)".
[part 1/5] Response to Reviewer Tn1r
Thank you for the detailed and rigorous review. We agree with the standard you set: the paper must show the value of the specific representation through controls, not only through end-to-end gains. In response, we:
Δ is given in percentage points relative to Full CSMem in Table 1.
W1. The added value of this specific organization (vs. A-MEM); RaMem. We agree that LLM-built structured memory is not new in itself. The question is what this particular organization adds, and we tested it directly: we fixed the retrieved episodes and the answering path, and replaced only the state.
The same construction model generates each substitute from the same payload, and the substitute is used wherever the state is used. End to end, A-MEM trails CSMem by 4.60 F1 / 5.50 EM / 5.16 BLEU-1 (Table 1).
RaMem-style control. As in RaMem, we held the memory content fixed, meaning the same retrieved episodes and the same raw-evidence package. We then shuffled the contextual record, which here is the state, across episodes of the same scenario. This removes 70% of the state's EM benefit (2.81 of 4.03), so what matters is that content and state are correctly bound.
Relation to RaMem. We share RaMem's premise that relevance is not the same as usefulness as evidence, and our Sec. 1 makes a closely related point.
The two approaches are complementary, and RaMem is text-only. We will discuss it in Related Work.