- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes S-MEM, a memory framework that organizes interaction histories by topic. Each Specialist stores interactions related to a specific topic and performs local reasoning, while the Master Agent selects relevant Specialists and combines their outputs. Experiments on LoCoMo and LongMemEval-S show improvements over existing memory methods.
Critical strengths
- The paper addresses a valuable problem of organizing long-term memory to support reasoning across multiple sessions.
- Organizing memory by topic allows related interactions from different sessions to be stored together, which may help cross-session reasoning.
Critical weaknesses
- The token consumption of S-MEM is much higher than that of other methods, but the improvement is marginal. Moreover, BLEU and F1 evaluate answers based on word matching, which can be easily influenced by unrelated words. The authors should report LLM Judge results.
- Each Specialist reads complete interaction blocks related to the same topic, so it may access more retrieved content than other methods, which could explain the performance improvement. The authors should report token consumption for different components to further explain this.
- Topics are created and assigned mainly through LLM prompts, without clear rules for when to merge or split topics.
- The main novelty of the paper is organizing memory based on different topics. Although the paper also introduces Specialist Agents and a Master Agent, it is unclear what advantages this design offers over directly organizing memory by topic, as explored in the related work.
Questions for authors
Please see the weaknesses above.
4 comments
Official Comment by Authors (2/2)
(c) Adaptive topic formation versus fixed topic buckets.
We additionally compare S-MEM with Fixed Topics, which assigns raw interaction blocks to predefined topic buckets instead of creating new Specialists incrementally. Both variants retain the same LLM backbone, Specialist local reasoning, and Master synthesis prompts; only the topic-formation strategy changes.
| LoCoMo variant | Overall F1 | BLEU-1 | ACC | Temporal ACC |
|---|---|---|---|---|
| Fixed Topics | 67.39 | 59.69 | 86.19 | 84.62 |
| Dynamic Topics (S-MEM) | 68.24 | 60.47 | 87.89 | 91.67 |
Fixed versus incremental topic formation with the local reasoning and synthesis pipeline unchanged.
Dynamic topic formation improves overall F1 by 0.85 points and judge accuracy by 1.70 percentage points, with a larger 7.05-point gain in temporal judge accuracy. The fixed-topic variant remains competitive, so*this experiment indicates an incremental benefit from adaptive topic formation, not that predefined topics are ineffective.
Together, these results indicate that S-MEM's performance gains cannot be attributed simply to higher token usage: comparable-budget E-MEM remains weaker, directly forwarding the same histories to the Master performs substantially worse, and adaptive topic formation improves performance with the reasoning pipeline held fixed. They support complementary roles for memory organization and Specialist-level processing, while not claiming that all systems read identical evidence or that the local-reasoning ablation isolates context-length effects. We will include these analyses and clarify their respective scopes in the revised manuscript.
3. Topic creation, merging, and splitting.(W3)
The need for topic merging and splitting depends on the application scenario. Our work focuses on long-term memory reasoning over finite multi-session interaction histories, where the primary challenge is connecting related information scattered across sessions, rather than managing continuously evolving topic structures. In this setting, incremental topic creation and assignment provide an effective mechanism for organizing memory, as supported by our experimental results.
We agree that explicit merging and splitting may become important in more complex, open-ended applications, where topics can gradually converge, diverge, or change in granularity over time. Extending S-MEM with adaptive topic restructuring for such lifelong scenarios is therefore a natural direction for future work.
4. Advantages of Specialist Agents beyond topic organization.(W4)
We thank the reviewer for highlighting the need to clarify the advantages of our agent-based design over existing topic-based memory approaches. Our core contribution is agent specialization, rather than topic-based memory organization itself. In S-MEM, each Specialist takes responsibility for a specific topic, continuously accumulating and reasoning over related interactions across sessions. To support this, we develop an integrated framework for Specialist registration, interaction allocation, query-driven activation, local reasoning, and cross-Specialist synthesis. Unlike conventional topic-based memory approaches that primarily organize information for storage and retrieval, S-MEM establishes persistent, specialized agents responsible for both maintaining and reasoning over topic-specific histories.
We further validate the effectiveness of this design through comparisons with existing memory approaches, including MemBox and AMA. Across three backbones on both benchmarks, S-MEM outperfomes both baselines in all three metrics, with gains of 3.07–3.88 percentage points on LoCoMo and 8.80–17.00 points on LongMemEval-S over the stronger baseline for each backbone.
| LoCoMo | GPT-4o-mini | Qwen3.5-27B | GPT-6.1-terra | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | F1 | BLEU‑1 | Acc. | F1 | BLEU‑1 | Acc. | F1 | BLEU‑1 | Acc. |
| MemBox | 60.80 | 54.81 | 78.45 | 65.52 | 59.89 | 83.53 | 57.61 | 52.62 | 75.58 |
| AMA | 63.83 | 58.11 | 82.88 | 65.73 | 60.83 | 82.63 | 68.99 | 60.40 | 84.39 |
| S-MEM | 67.57 | 60.00 | 85.95 | 69.60 | 62.19 | 87.41 | 70.62 | 62.69 | 87.89 |
| LongMemEval-S | GPT-4o-mini | Qwen3.5-27B | GPT-6.1-terra | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | F1 | BLEU‑1 | ACC | F1 | BLEU‑1 | ACC | F1 | BLEU‑1 | ACC |
| MemBox | 43.30 | 34.29 | 62.40 | 48.60 | 38.93 | 71.20 | 50.13 | 39.41 | 79.00 |
| AMA | 46.58 | 36.95 | 65.20 | 51.25 | 42.83 | 75.20 | 54.15 | 43.96 | 81.60 |
| S-MEM | 57.53 | 50.39 | 82.20 | 61.29 | 57.59 | 88.80 | 63.46 | 60.07 | 90.40 |
Thank you for the detailed response and the additional experiments. The LLM-as-judge results address my concern about lexical metrics. However, several concerns remain.
-
In Table 10, S-MEM consumes more than three times the most expensive competing method. The budget-matched experiment raises the cost of E-MEM, but this does not resolve the issue. It only shows that E-MEM does not benefit from additional computation. Moreover, the cost on LongMemEval-S, where histories are much longer, is not reported.
-
The response states that this work focuses on finite multi-session histories rather than continuously evolving interactions. However, long-term memory systems such as A-MEM are designed to incrementally accommodate interaction histories of growing length, and supporting continuously growing histories is a central requirement of agent memory. Restricting the method to finite histories therefore substantially limits its applicability.
-
The paper repeatedly positions its novelty over E-MEM as organizing memory agents by topic rather than by time, whereas the response claims the core contribution is agent specialization rather than topic-based organization. However, a similar master and memory-agent framework already exists in E-MEM. Therefore, the novelty of the work remains unclear.
Thank you for the follow-up questions. We clarify the scope of the budget-matched comparison, how S-MEM handles growing histories, and what distinguishes it from E-MEM.
1.Token cost and LongMemEval-S
We agree that the budget-matched comparison does not resolve S-MEM’s higher token cost under the default settings. Reducing the query-time token cost of Specialist reasoning is a priority for future work. We have updated the Limitations section to discuss this trade-off. In particular, we will investigate how to reduce the amount of history each Specialist processes per query while retaining the evidence needed for cross-session reasoning. We did not record token usage during the original LongMemEval-S evaluations and could not rerun the full comparison within the rebuttal window. For example, a full A-MEM evaluation on LongMemEval-S takes more than three days in our setup. We will add the LongMemEval-S token breakdown to the paper in the future.
2. Incremental updates to growing histories
Sorry for the confusion. Our earlier reference to “finite histories” was imprecise. S-MEM does not require a fixed-length or complete interaction history. Its write path is incremental. The finite histories describe the evaluation setting, not an input restriction. The current system can continue to ingest new interactions. As new blocks arrive, the Allocator appends them to relevant Specialists and creates new Specialists for uncovered topics without rebuilding existing memory.
3. Contribution relative to E-MEM
E-MEM already uses a Master and locally reasoning memory agents. S-MEM changes the scope of local reasoning from an individual episode to a topic history that grows across sessions. In E-MEM, evidence distributed across episodes is combined by the Master. In S-MEM, a Specialist can connect related evidence from different sessions within its own reasoning context before sending its findings to the Master for cross-topic synthesis. Topic-based organization is therefore what enables Specialist specialization, not an alternative explanation of the contribution.
Even with S-MEM's reasoning pipeline held fixed, E-MEM-style episode histories are less effective than topic histories. In the within-S-MEM ablation reported in W2(c), replacing topic histories with episode histories lowers LoCoMo overall F1 from 67.39 to 63.10 and LLM-as-judge accuracy from 86.19% to 79.18%.
Official Comment by Authors (1/2)
We thank the reviewer for these constructive comments. Below, we provide LLM-as-judge results, examine performance under comparable token budgets, clarify component-wise computation, and distinguish the contributions of topic organization and Specialist reasoning.
1. Token cost versus performance; LLM-as-judge evaluation.(W1)
The gains are not limited to word-overlap metrics, and additional tokens alone do not explain them.
(a) LLM-as-judge.
We evaluated answers with an LLM judge, using the same final-answer prompt template and judge prompt across methods within each benchmark/backbone setting. S-MEM attains the highest LLM-as-judge accuracy for all three evaluated backbones on both benchmarks. On LongMemEval-S, it exceeds A-MEM—the strongest baseline—by 5.00, 6.20, and 4.00 points, respectively. On LoCoMo, its gains over the strongest competing LLM-as-judge result for each backbone are 2.26, 3.88, and 2.10 points.
(b) Comparable query-time token budgets.
S-MEM outperforms both E-MEM configurations at similar total token use (within 4.3% of S-MEM's budget). We increased E-MEM's reasoning budget on LoCoMo to test whether S-MEM's advantage simply comes from more computation.
2×8means two reasoning rounds with up to eight memory agents per round;3×5means three rounds with up to five.Relative to the stronger E-MEM 2×8 configuration, it gains 4.00 F1, 5.62 BLEU-1, and 4.14 LLM-as-judge accuracy points. This does not establish equal efficiency against every baseline, but it shows that raising E-MEM's query-time budget to a comparable level does not close the performance gap.
2. Specialist context access and component-wise token consumption.(W2)
We agree that accessing more context may contribute to S-MEM's performance gains. To examine whether the improvement is explained solely by greater token consumption or by the organization and processing of retained histories, we provide three complementary analyses.
(a) Component-wise token consumption and a comparable-budget comparison.
We break down query-time tokens into Master and memory-agent/Specialist consumption and increase E-MEM's reasoning budget to approach S-MEM's. The table are shown in W1(b). Specialists account for 90.2% of S-MEM's query-time tokens. Although E-MEM's agent-side consumption is within 2.4% of S-MEM's in both configurations, S-MEM still exceeds the stronger E-MEM (2 × 8) in all three metrics. Thus, greater aggregate token consumption alone does not account for this performance gap. Table 10's original S-MEM total mixed query-time inference with two costs that should have been reported separately. We have corrected the accounting.
(b) Specialist local reasoning versus direct history forwarding.
In the original ablation (Section 4.3), the same topic-organized histories are forwarded directly to the Master instead of being processed by Specialists. On full LoCoMo with a GPT-4o-mini Master, removing local reasoning reduces overall F1 from 67.57 to 47.09, multi-hop F1 from 47.61 to 28.46, and temporal F1 from 61.95 to 39.18.
Effect of replacing Specialist processing with direct forwarding of the same retained topic histories (GPT-4o-mini Master). This shows that access to the original histories alone is not sufficient under the tested direct-read configuration: how the histories are processed matters. Because direct forwarding also concentrates longer raw contexts at the Master, the result supports the hierarchical processing design as a whole, rather than isolating reasoning independently of context length.