- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focuses on factual updates and procedural evolution in persistent memory for LLM agents. The authors formalize a shared experience representation and propose distinct update mechanisms for semantic and procedural memory. Semantic updates revise facts based on entity identity, scope, and observation time while retaining history and unresolved conflicts. Procedural updates derive reusable steps from successful trajectories and use failures to refine applicability conditions and reuse eligibility. Both memory types remain linked to shared experience scenes and source evidence.
Critical strengths
- Evidence-aware memory management abstraction is clear: preserves provenance, history, conflicts, and skill applicability.
- Ablation studies and controlled comparisons of this paper is solid.
Critical weaknesses
- The gain attribution remains unclear: extra calculation, execution repair, and BFCL packing/deletion safeguards may contribute. Reported configurations, baseline parity, and τ-bench feedback permissions remain unspecified.
- Abstract and introduction list mechanisms without establishing why existing methods fail or shared experience is necessary. No matched unified-versus-distinct update comparison tests the central claim.
- Only two model is used in evaluation for -bench, more model should be included to prove the method can be generalied.
Questions for authors
- Which -bench feedback fields enter memory updates? Are reference actions and hidden evaluator information excluded, with identical feedback access across methods?
- With calculation, execution checks, and repairs matched across methods, how much improvement remains?
- Do BFCL baselines receive identical packing, deletion safeguards, and retrieval budgets? What gains remain after matching them?
- With representation and retrieval fixed, do distinct update criteria outperform a unified outcome-based rule? Also, only use dsv4 as judge is too limited.
2 comments
Thank you for the detailed response and additional experiments. However, my main concerns remain only partially resolved.
-
the proposed comparison against a unified success-gated update rule is particularly important, as it directly tests whether the distinct semantic and procedural update criteria are necessary. The existing comparisons do not fully isolate this contribution.
-
the BFCL comparison with matched packing, deletion protection, and retrieval budgets is still pending. Similarly, the Full vs. Frozen experiment demonstrates the benefit of continued memory updates but does not fully isolate the contribution of auxiliary computation, execution checking, and repair.
-
I would appreciate explicit clarification of which τ-bench feedback fields are available for memory updates, whether any hidden evaluator information is excluded, and whether all baselines have equivalent feedback access.
The additional judge evaluation is reassuring, although it does not directly address my suggestion of a dsv4-only evaluation. The generalization evidence also remains limited to two agent backbones.
I appreciate the authors' efforts, but I would like to see the promised controlled comparisons before reconsidering my assessment.
Thank you for the comments on gain attribution, evaluation conditions, and generalization. We will strengthen the controlled comparisons and clarify the evaluation protocol.
Distinct update criteria and shared experience.
Factual revision and procedural learning require different evidence from the same interaction. An unsuccessful task can establish a factual change, while reusing a procedure requires evidence that its steps were effective under compatible conditions. Shared scenes connect the context and execution evidence supporting these decisions.
With Qwen3.8-27B, full SEEM² achieves 80.22% task success, compared with 74.82% for separate memory organization and 76.98% without updates from failed tasks. We will add a comparison with a unified task-success gate for positive semantic and procedural updates, holding initial memory, representation, retrieval, backbone, and task order fixed. We will report overall and per-domain success.
Auxiliary calculation and continuous updates.
A supplementary fixed-memory evaluation without auxiliary calculation achieves 75.00% on LongMemEval S∗ with Gemma4 and 78.33% with DeepSeek-V4-Flash, using one answer-generation call per question.
The paper’s Qwen3.8-27B Full/Frozen comparison starts from the same initial memory and uses matched task order, backbone, retrieval, and task-execution settings. Full achieves 81.53% success over 222 tasks versus 74.32% for Frozen. Memory updates occur after task scoring, using completed visible trajectories and outcome feedback. We will list the update-feedback fields and document the access available to each baseline in the appendix.
BFCL protocol and token cost.
We follow the official BFCL V4
memory_vectorprotocol: 37 prerequisite conversations followed by 155 evaluation questions, using the official memory-tool interface, capacity limits, and answer-matching checker.Our completed cost audit gives:
\text{Method} & \text{Accuracy} & \text{Total LLM tokens} \\\\ \hline \mathrm{SEEM}^{2} & \mathbf{92.90\\%} & 10.22\mathrm{M} \\\\ \text{Mem0} & 75.48\\% & 12.84\mathrm{M} \\\\ \text{Graphiti} & 74.84\\% & 16.01\mathrm{M} \\\\ \text{LightMem} & 74.19\\% & 9.37\mathrm{M} \\\\ \text{Vector Store} & 56.77\\% & 6.53\mathrm{M} \end{array}$$ These totals cover construction and question answering, excluding embeddings. SEEM² achieves higher accuracy with fewer total tokens than Mem0 and Graphiti; LightMem and Vector Store use fewer tokens. To assess the contribution of packing, deletion protection, and retrieval budgets, we will add a comparison between SEEM² and Mem0 with these settings matched and report accuracy and token cost. **Evaluation with another judge.** We have reevaluated the main paper’s saved SEEM² predictions and all eight earlier baseline prediction sets with Gemma4, keeping predictions fixed. The table also includes the newly evaluated original SEEM baseline. $$\begin{array}{lr} \text{Method} & \text{Gemma4 accuracy} \\\\ \hline \mathrm{SEEM}^{2} & \mathbf{75.33\\%} \\\\ \text{Long-context FIFO} & 66.00\\% \\\\ \text{Original SEEM} & 65.67\\% \\\\ \text{Dense RAG, Qwen3-Embedding-8B} & 65.33\\% \\\\ \text{Graphiti} & 64.67\\% \\\\ \text{Dense RAG, KaLM} & 63.00\\% \\\\ \text{Full Context} & 61.00\\% \\\\ \text{LightMem} & 59.67\\% \\\\ \text{A-MEM} & 52.00\\% \\\\ \text{Mem0} & 51.33\\% \end{array}$$ Each row covers all 300 questions. SEEM² remains the highest-scoring method under Gemma4, and the two judges agree on 286 of its 300 predictions. We will include this evaluation in the revision. **Generalization across agent backbones.** SEEM² achieves 93.17% task success with DeepSeek-V4-Flash versus 90.65% for the strongest evaluated baseline, and 80.22% with Qwen3.8-27B versus 74.46%. We will extend the $\tau$-bench evaluation to an additional agent backbone.