acceptodds
Under review as a conference paper at ICLR 2027

S-MEM: Specialized Memory Agents for Long-term Memory of LLM Agents

Abstract

Long-term memory is essential for LLM agents operating over extended interactions, where answering a query may require connecting evidence scattered across sessions. Existing memory systems typically store past interactions in external structures, such as vector databases or knowledge graphs, which can fragment original interaction contexts and weaken contextual dependencies for long-horizon reasoning. Recent memory-agent approaches instead use dedicated agents whose contexts directly serve as memory, preserving interaction histories for local reasoning. However, they typically organize memory by individual interaction episodes, causing related experiences from different sessions to be stored in separate agents and making cross-session reasoning difficult. We propose Specialized Memory Agents (S-MEM), a long-term memory framework that organizes memory agents by topic rather than by interaction episode. Each Specialist is dedicated to a specific topic and maintains the interaction history related to that topic across sessions. Given a query, the Master Agent activates relevant Specialists, which reason over their accumulated histories, and synthesizes their evidence for global reasoning. By consolidating related experiences within the same memory agent, S-MEM enables cross-session reasoning while preserving the contextual richness of original interactions. Experiments on LoCoMo and LongMemEval show that S-MEM consistently improves F1 and BLEU-1 over strong baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, long-term memory

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_wHZAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S-MEM, a memory framework that organizes interaction histories by topic. Each Specialist stores interactions related to a specific topic and performs local reasoning, while the Master Agent selects relevant Specialists and combines their outputs. Experiments on LoCoMo and LongMemEval-S show improvements over existing memory methods.

Critical strengths
  1. The paper addresses a valuable problem of organizing long-term memory to support reasoning across multiple sessions.
  2. Organizing memory by topic allows related interactions from different sessions to be stored together, which may help cross-session reasoning.
Critical weaknesses
  1. The token consumption of S-MEM is much higher than that of other methods, but the improvement is marginal. Moreover, BLEU and F1 evaluate answers based on word matching, which can be easily influenced by unrelated words. The authors should report LLM Judge results.
  2. Each Specialist reads complete interaction blocks related to the same topic, so it may access more retrieved content than other methods, which could explain the performance improvement. The authors should report token consumption for different components to further explain this.
  3. Topics are created and assigned mainly through LLM prompts, without clear rules for when to merge or split topics.
  4. The main novelty of the paper is organizing memory based on different topics. Although the paper also introduces Specialist Agents and a Master Agent, it is unclear what advantages this design offers over directly organizing memory by topic, as explored in the related work.
Questions for authors

Please see the weaknesses above.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for these constructive comments. Below, we provide LLM-as-judge results, examine performance under comparable token budgets, clarify component-wise computation, and distinguish the contributions of topic organization and Specialist reasoning.

1. Token cost versus performance; LLM-as-judge evaluation.(W1)

The gains are not limited to word-overlap metrics, and additional tokens alone do not explain them.

(a) LLM-as-judge.

We evaluated answers with an LLM judge, using the same final-answer prompt template and judge prompt across methods within each benchmark/backbone setting. S-MEM attains the highest LLM-as-judge accuracy for all three evaluated backbones on both benchmarks. On LongMemEval-S, it exceeds A-MEM—the strongest baseline—by 5.00, 6.20, and 4.00 points, respectively. On LoCoMo, its gains over the strongest competing LLM-as-judge result for each backbone are 2.26, 3.88, and 2.10 points.

LongMemEval-SGPT-4o-miniQwen3.5-27BGPT-6.1-terra
Full Context55.4063.8080.40
Naïve RAG74.2071.2070.20
LightMem57.4069.4082.20
GAM48.6074.2076.40
A-MEM77.2082.6086.40
MemBox62.4071.2079.00
AMA65.2075.2081.60
E-MEM71.2081.8083.00
S-MEM82.2088.8090.40
LoCoMoGPT-4o-miniQwen3.5-27BGPT-6.1-terra
Full Context77.8482.4883.78
Naïve RAG65.4171.8071.67
LightMem76.6478.6079.47
GAM74.7782.9885.79
A-MEM72.2176.0379.48
MemBox78.4583.5375.58
AMA82.8882.6384.39
E-MEM83.6983.4881.38
S-MEM85.9587.4187.89

(b) Comparable query-time token budgets.

S-MEM outperforms both E-MEM configurations at similar total token use (within 4.3% of S-MEM's budget). We increased E-MEM's reasoning budget on LoCoMo to test whether S-MEM's advantage simply comes from more computation. 2×8 means two reasoning rounds with up to eight memory agents per round; 3×5 means three rounds with up to five.

ConfigurationTotal tokens / questionMaster tokensMemory-agent tokensF1BLEU-1ACC
E-MEM 2×8107,834.918,284.2499,550.6763.5754.3881.81
E-MEM 3×5110,015.6410,792.8599,222.7962.7453.8380.64
S-MEM112,584.2210,988.02101,596.2067.5760.0085.95

Relative to the stronger E-MEM 2×8 configuration, it gains 4.00 F1, 5.62 BLEU-1, and 4.14 LLM-as-judge accuracy points. This does not establish equal efficiency against every baseline, but it shows that raising E-MEM's query-time budget to a comparable level does not close the performance gap.

2. Specialist context access and component-wise token consumption.(W2)

We agree that accessing more context may contribute to S-MEM's performance gains. To examine whether the improvement is explained solely by greater token consumption or by the organization and processing of retained histories, we provide three complementary analyses.

(a) Component-wise token consumption and a comparable-budget comparison.

We break down query-time tokens into Master and memory-agent/Specialist consumption and increase E-MEM's reasoning budget to approach S-MEM's. The table are shown in W1(b). Specialists account for 90.2% of S-MEM's query-time tokens. Although E-MEM's agent-side consumption is within 2.4% of S-MEM's in both configurations, S-MEM still exceeds the stronger E-MEM (2 × 8) in all three metrics. Thus, greater aggregate token consumption alone does not account for this performance gap. Table 10's original S-MEM total mixed query-time inference with two costs that should have been reported separately. We have corrected the accounting.

(b) Specialist local reasoning versus direct history forwarding.

In the original ablation (Section 4.3), the same topic-organized histories are forwarded directly to the Master instead of being processed by Specialists. On full LoCoMo with a GPT-4o-mini Master, removing local reasoning reduces overall F1 from 67.57 to 47.09, multi-hop F1 from 47.61 to 28.46, and temporal F1 from 61.95 to 39.18.

LoCoMo variantOverall F1Multi-hop F1Temporal F1
S-MEM67.5747.6161.95
w/o Specialist local reasoning47.0928.4639.18

Effect of replacing Specialist processing with direct forwarding of the same retained topic histories (GPT-4o-mini Master). This shows that access to the original histories alone is not sufficient under the tested direct-read configuration: how the histories are processed matters. Because direct forwarding also concentrates longer raw contexts at the Master, the result supports the hierarchical processing design as a whole, rather than isolating reasoning independently of context length.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

(c) Adaptive topic formation versus fixed topic buckets.

We additionally compare S-MEM with Fixed Topics, which assigns raw interaction blocks to predefined topic buckets instead of creating new Specialists incrementally. Both variants retain the same LLM backbone, Specialist local reasoning, and Master synthesis prompts; only the topic-formation strategy changes.

LoCoMo variantOverall F1BLEU-1ACCTemporal ACC
Fixed Topics67.3959.6986.1984.62
Dynamic Topics (S-MEM)68.2460.4787.8991.67

Fixed versus incremental topic formation with the local reasoning and synthesis pipeline unchanged.

Dynamic topic formation improves overall F1 by 0.85 points and judge accuracy by 1.70 percentage points, with a larger 7.05-point gain in temporal judge accuracy. The fixed-topic variant remains competitive, so*this experiment indicates an incremental benefit from adaptive topic formation, not that predefined topics are ineffective.

Together, these results indicate that S-MEM's performance gains cannot be attributed simply to higher token usage: comparable-budget E-MEM remains weaker, directly forwarding the same histories to the Master performs substantially worse, and adaptive topic formation improves performance with the reasoning pipeline held fixed. They support complementary roles for memory organization and Specialist-level processing, while not claiming that all systems read identical evidence or that the local-reasoning ablation isolates context-length effects. We will include these analyses and clarify their respective scopes in the revised manuscript.

3. Topic creation, merging, and splitting.(W3)

The need for topic merging and splitting depends on the application scenario. Our work focuses on long-term memory reasoning over finite multi-session interaction histories, where the primary challenge is connecting related information scattered across sessions, rather than managing continuously evolving topic structures. In this setting, incremental topic creation and assignment provide an effective mechanism for organizing memory, as supported by our experimental results.

We agree that explicit merging and splitting may become important in more complex, open-ended applications, where topics can gradually converge, diverge, or change in granularity over time. Extending S-MEM with adaptive topic restructuring for such lifelong scenarios is therefore a natural direction for future work.

4. Advantages of Specialist Agents beyond topic organization.(W4)

We thank the reviewer for highlighting the need to clarify the advantages of our agent-based design over existing topic-based memory approaches. Our core contribution is agent specialization, rather than topic-based memory organization itself. In S-MEM, each Specialist takes responsibility for a specific topic, continuously accumulating and reasoning over related interactions across sessions. To support this, we develop an integrated framework for Specialist registration, interaction allocation, query-driven activation, local reasoning, and cross-Specialist synthesis. Unlike conventional topic-based memory approaches that primarily organize information for storage and retrieval, S-MEM establishes persistent, specialized agents responsible for both maintaining and reasoning over topic-specific histories.

We further validate the effectiveness of this design through comparisons with existing memory approaches, including MemBox and AMA. Across three backbones on both benchmarks, S-MEM outperfomes both baselines in all three metrics, with gains of 3.07–3.88 percentage points on LoCoMo and 8.80–17.00 points on LongMemEval-S over the stronger baseline for each backbone.

LoCoMoGPT-4o-miniQwen3.5-27BGPT-6.1-terra
MethodF1BLEU‑1Acc.F1BLEU‑1Acc.F1BLEU‑1Acc.
MemBox60.8054.8178.4565.5259.8983.5357.6152.6275.58
AMA63.8358.1182.8865.7360.8382.6368.9960.4084.39
S-MEM67.5760.0085.9569.6062.1987.4170.6262.6987.89
LongMemEval-SGPT-4o-miniQwen3.5-27BGPT-6.1-terra
MethodF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACC
MemBox43.3034.2962.4048.6038.9371.2050.1339.4179.00
AMA46.5836.9565.2051.2542.8375.2054.1543.9681.60
S-MEM57.5350.3982.2061.2957.5988.8063.4660.0790.40

Thank you for the detailed response and the additional experiments. The LLM-as-judge results address my concern about lexical metrics. However, several concerns remain.

  1. In Table 10, S-MEM consumes more than three times the most expensive competing method. The budget-matched experiment raises the cost of E-MEM, but this does not resolve the issue. It only shows that E-MEM does not benefit from additional computation. Moreover, the cost on LongMemEval-S, where histories are much longer, is not reported.

  2. The response states that this work focuses on finite multi-session histories rather than continuously evolving interactions. However, long-term memory systems such as A-MEM are designed to incrementally accommodate interaction histories of growing length, and supporting continuously growing histories is a central requirement of agent memory. Restricting the method to finite histories therefore substantially limits its applicability.

  3. The paper repeatedly positions its novelty over E-MEM as organizing memory agents by topic rather than by time, whereas the response claims the core contribution is agent specialization rather than topic-based organization. However, a similar master and memory-agent framework already exists in E-MEM. Therefore, the novelty of the work remains unclear.

Thank you for the follow-up questions. We clarify the scope of the budget-matched comparison, how S-MEM handles growing histories, and what distinguishes it from E-MEM.

1.Token cost and LongMemEval-S

We agree that the budget-matched comparison does not resolve S-MEM’s higher token cost under the default settings. Reducing the query-time token cost of Specialist reasoning is a priority for future work. We have updated the Limitations section to discuss this trade-off. In particular, we will investigate how to reduce the amount of history each Specialist processes per query while retaining the evidence needed for cross-session reasoning. We did not record token usage during the original LongMemEval-S evaluations and could not rerun the full comparison within the rebuttal window. For example, a full A-MEM evaluation on LongMemEval-S takes more than three days in our setup. We will add the LongMemEval-S token breakdown to the paper in the future.

2. Incremental updates to growing histories

Sorry for the confusion. Our earlier reference to “finite histories” was imprecise. S-MEM does not require a fixed-length or complete interaction history. Its write path is incremental. The finite histories describe the evaluation setting, not an input restriction. The current system can continue to ingest new interactions. As new blocks arrive, the Allocator appends them to relevant Specialists and creates new Specialists for uncovered topics without rebuilding existing memory.

3. Contribution relative to E-MEM

E-MEM already uses a Master and locally reasoning memory agents. S-MEM changes the scope of local reasoning from an individual episode to a topic history that grows across sessions. In E-MEM, evidence distributed across episodes is combined by the Master. In S-MEM, a Specialist can connect related evidence from different sessions within its own reasoning context before sending its findings to the Master for cross-topic synthesis. Topic-based organization is therefore what enables Specialist specialization, not an alternative explanation of the contribution.

Even with S-MEM's reasoning pipeline held fixed, E-MEM-style episode histories are less effective than topic histories. In the within-S-MEM ablation reported in W2(c), replacing topic histories with episode histories lowers LoCoMo overall F1 from 67.39 to 63.10 and LLM-as-judge accuracy from 86.19% to 79.18%.

Reviewer_6mXmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes S-MEM, a long-term memory system that organizes memory agents by persistent topics rather than by interaction episodes. An allocator assigns conversation blocks to topic-specific Specialists. At query time, a Master agent activates relevant Specialists, which reason over their accumulated histories and return evidence for final synthesis. The paper evaluates the system on LoCoMo and LongMemEval-S.

Critical strengths

The paper presents a clear motivation: information about the same topic can recur across sessions, making episode-based organization inconvenient for cross-session reasoning. The method is easy to understand, and the evaluation includes two benchmarks, two Master backbones, and a direct comparison with E-MEM. S-MEM obtains the best overall F1 in the reported main tables, with particularly useful gains over E-MEM on LoCoMo.

Critical weaknesses
  1. The novelty needs sharper positioning. The main change from E-MEM is to organize memory agents by topic instead of episode. Topic-based organization of dialogue memory has also been explored previously; for example, RMM (https://aclanthology.org/2025.acl-long.413.pdf) organizes raw dialogue around topics and merges related memories across sessions. RMM does not use topic-specific reasoning agents, so it does not subsume S-MEM. However, the paper should explain more precisely what benefit comes from combining topic organization with local agent reasoning.
  2. The central design choice lacks a decisive control. A topic-organized memory store with a single reader, or an E-MEM variant that groups episodes by topic, would help isolate the contribution of persistent Specialists. The reported “without local reasoning” ablation passes Specialist histories directly to the Master. Its large performance drop may partly reflect the difficulty of processing much more context in one place; it does not by itself establish that the proposed agent structure is the best way to use the same information and compute.
  3. The gains should be assessed against their cost. On LongMemEval-S, the overall F1 gains over the strongest baseline are 0.88 and 0.46 percentage points for the two Master backbones, although the BLEU-1 gains are larger. Appendix E reports approximately 113,389 tokens per query for S-MEM versus 31,383 for E-MEM on LoCoMo. A comparison at matched inference budgets, or a performance–cost curve, is needed to judge whether the improvement justifies this expense.
  4. Some analyses are too limited to support broad claims. The allocation analysis uses the first LoCoMo conversation, while the multi-Specialist assignment ablation uses the first two. The paper also does not report uncertainty for the main performance differences. Finally, because each Specialist retains full assigned blocks and blocks may be duplicated across Specialists, it would be useful to measure how memory size and per-query context cost grow over longer histories.
Questions for authors
  • How does S-MEM compare with topic-organized memory without local Specialist reasoning under a matched token budget?
  • How do the number of Specialists, duplicated stored tokens, and query-time token use scale as more sessions are added?
  • Why use token-level F1 on LongMemEval-S instead of the official LLM-as-judge evaluation? Could the small F1 gains reflect prompt-induced answer-format alignment rather than better semantic correctness? Do the gains persist under matched prompts and judge-based evaluation?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the constructive comments. We have strengthened the novelty discussion and added controlled comparisons, budget-matched experiments, full-benchmark analyses, uncertainty estimates, and a scaling study. We address the concerns below.

1. Novelty relative to existing topic-oriented memory methods (W1)

We thank the reviewer for this comment. While topic-oriented methods like RMM and MemBox already organize conversational memories by topic, S-MEM differs in how these memories are used for reasoning. Existing methods organize and retrieve topic-related memories for a central LLM to answer queries. In contrast, S-MEM maintains persistent topic-specific Specialists that accumulate related interactions across sessions and perform local reasoning over their complete histories before returning evidence and conclusions to the Master. Thus, S-MEM extends topic-based memory organization to topic-based reasoning, enabling cross-session evidence to be jointly interpreted within each Specialist rather than relying solely on retrieved memories. We will clarify this distinction in the Introduction and Related Work.

2. Controlled comparisons of topic organization and Specialist reasoning (W2)

We thank the reviewer for pointing out that our original without local reasoning ablation may confound the contribution of Specialists with the difficulty of reading long histories in a single Master context. To address this concern, we add a comparison with MemBox (Trace), a topic-organized memory system that retrieves selected evidence for a single QA reader rather than passing entire histories to that reader. We additionally compare dynamic and fixed topic formation under the same reasoning pipeline to distinguish the value of adaptive organization from the system-level MemBox comparison.

(a) Dynamic versus fixed topic organization.

We compare S-MEM with Fixed Topics, which assigns raw conversation blocks to predefined topic buckets but cannot create new ones. Both variants use the same LLM backbone, Specialist local reasoning, and Master synthesis prompts. On LoCoMo, dynamic organization improves overall F1 from 67.39 to 68.24 (+0.85 points) and LLM-as-judge accuracy from 86.19% to 87.89% (+1.70 percentage points), isolating the incremental benefit of adaptive topic formation within the same reasoning architecture.

LoCoMoOverall F1BLEU-1ACCTemporal ACC
Fixed Topics67.3959.6986.1984.62
S-MEM68.2460.4787.8991.67

(b) Comparison with a topic-organized single-reader method.

MemBox (Trace) groups topic-coherent interactions into memory boxes and links related events across sessions. At query time, it retrieves relevant boxes and traces for a single QA reader. Across all three Master backbones on LoCoMo, S-MEM outperforms MemBox in overall F1 by 4.08–13.01 points and in overall judge accuracy by 3.88–12.31 percentage points. It also improves multi-hop F1 by 5.27–10.06 points.

LoCoMo — S-MEM versus MemBox (Trace)

BackboneMethodOverallMulti-HopTemporalOpen DomainSingle-HopAdversarial
F1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACC
GPT-4o-miniMemBox60.8054.8178.4540.4829.7969.1560.9546.2377.2629.7923.0452.0860.0152.7983.4780.9980.9781.39
S-MEM67.5760.0085.9547.6139.6770.2161.9547.9084.4230.7524.0576.6764.9255.9187.9997.1697.0197.31
Qwen3.5-27bMemBox65.5259.8983.5342.9132.9475.1862.7351.9777.5725.0519.8545.8363.1055.9586.6895.0795.0795.29
S-MEM69.6062.1987.4148.1838.5279.5065.5153.0085.9838.6031.7772.9266.3958.0989.5498.0898.0598.88
GPT-6.1-terraMemBox57.6152.6275.5842.0533.9864.0864.9858.1685.9023.3318.5950.0060.6454.5585.4163.3863.3463.29
S-MEM70.6262.6987.8952.1143.0279.0172.8161.3292.3135.3029.5070.2264.9956.1687.0898.8398.6298.89

Together, these results supports the benefit of adaptive topic formation of S-MEM. We will include these results and clarify their scope in the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

3. Performance at comparable inference budgets (W3)

(a) Performance at comparable token budgets on LoCoMo.

S-MEM outperformed both higher-budget E-MEM configurations on all three metrics, even when their average query-time token use was within 4.3% of S-MEM’s. Compared with the better-performing E-MEM configuration (2×8), S-MEM gained 4.00 F1 points and 4.14 percentage points in LLM-judge accuracy.

The original E-MEM configuration uses one reasoning round with up to five memory agents (1×5). To bring E-MEM’s token usage closer to S-MEM’s, we increased either the number of rounds alone (3×5) or both the number of rounds and the per-round activation limit (2×8).

ConfigurationTotal tokens / questionMaster tokens / questionMemory agent tokens / questionF1BLEU-1ACC
E-MEM 2×8107,834.918,284.2499,550.6763.5754.3881.81
E-MEM 3×5110,015.6410,792.8599,222.7962.7453.8380.64
S-MEM112,584.2210,988.02101,596.2067.5760.0085.95

We also updated some details in original Table 10 accordingly.

(b) LLM-as-judge evaluation on LongMemEval-S.

LLM-as-judge evaluation on LongMemEval-S. We added LLM-as-judge accuracy to complement F1 and BLEU-1. On this metric, S-MEM shows a larger lead over A-MEM than the modest F1 gains observed with the two original backbones. We also added Membox and AMA as baselines and GPT-6.1-terra as a third Master backbone.

MethodGPT-4o-miniQwen3.5-27BGPT-6.1-terra
Full Context55.4063.8080.40
Naïve RAG74.2071.2070.20
LightMem57.4069.4082.20
GAM48.6074.2076.40
A-MEM77.2082.6086.40
Membox62.4071.2079.00
AMA65.2075.2081.60
E-MEM71.2081.8083.00
S-MEM82.2088.8090.40

4.Expanded evaluation and scalability (W4 & Q2)

(a) Specialist allocation.

We extended the allocation analysis to the full LoCoMo benchmark. The full-benchmark results confirm our main finding from the first conversation that evidence reachability remains high across Allocator backbones despite differences in routing behavior.

AllocatorReach.PRRoutes
GPT-4o-mini97.9994.5272.831.94
Qwen3.5-27B98.5991.7868.702.24
DeepSeek-V4-Flash98.9493.7288.812.74

(b) Multi-Specialist assignment.

We extended the assignment ablation from the first two conversations to the full LoCoMo benchmark and updated Table 7 in the revised manuscript. Multi-Specialist assignment improves all three metrics in most category. This pattern is consistent with our explanation in §4.3.

AssignmentOverallMulti-HopTemporalOpen DomainSingle-HopAdversarial
F1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACC
Primary-only62.9355.6880.2846.8738.2858.1159.9445.3581.1127.5916.3775.7151.1542.9577.5097.9897.5598.21
S-MEM67.5760.0085.9547.6139.6770.2161.9547.9084.4230.7524.0576.6764.9255.9187.9997.1697.0197.31

(c) Uncertainty in main performance differences.

We estimated 95% confidence intervals for the differences between S-MEM and A-MEM, the strongest baseline on LongMemEval-S. With GPT-4o-mini, S-MEM improves F1 by 0.88 points (95% CI [0.10, 2.50]) and BLEU-1 by 5.41 points (95% CI [4.29, 6.65]). With Qwen3.5-27B, the gains are 0.46 F1 points (95% CI [0.25, 2.40]) and 6.47 BLEU-1 points (95% CI [5.16, 8.02]). All four intervals lie above zero, supporting positive gains under the uncertainty captured by this analysis.

(d) Memory and query costs as sessions accumulate.(Q2)

From LoCoMo, we selected three conversations with more than 30 sessions and measured memory size and query-time token usage at six checkpoints from 5 to 30 ingested sessions. Overall, storage and query-time token usage grew approximately linearly over the measured range.

Ingested sessionsAvg. number of SpecialistsTotal stored tokensDuplicated storage tokensAvg. query-time tokens per question
53.0013,6028,30131,963
103.3326,64016,51651,488
153.3340,94125,49179,396
203.3355,90835,273106,835
253.3371,15945,502129,699
303.3386,44055,742156,791
AuthorsOpenReview ↗

Official Comment by Authors (3/3)

5. Topic-organized memory without local reasoning (Q1)

We agree that a strictly token-matched, optimized topic-organized single-reader baseline remains an important control. Our additional matched-backbone ablation compares the same topic histories under two processing strategies on the first LoCoMo conversation: direct reading by the Master versus local Specialist processing followed by Master synthesis.

VariantTotal tokens/queryMaster tokens/querySpecialist tokens/queryF1BLEU-1ACC
w/o local reasoning124,671.61124,671.61058.0251.7186.93
S-MEM91,322.9610,553.6880,769.2866.6159.3491.46

S-MEM improves F1 by 8.59 points and judge accuracy by 4.53 percentage points, while using 26.7% fewer total query-time tokens.

6. LLM-as-judge protocol and answer-format effects (Q3)

We added LLM-as-judge accuracy alongside lexical metrics and used the same final-answer prompt template and judge prompt for all compared methods under each backbone.

LongMemEval-SGPT-4o-miniQwen3.5-27BGPT-6.1-terra
Full Context55.4063.8080.40
Naïve RAG74.2071.2070.20
LightMem57.4069.4082.20
GAM48.6074.2076.40
A-MEM77.2082.6086.40
MemBox (Trace)62.4071.2079.00
AMA65.2075.2081.60
E-MEM71.2081.8083.00
S-MEM82.2088.8090.40
LoCoMoGPT-4o-miniQwen3.5-27BGPT-6.1-terra
Full Context77.8482.4883.78
Naïve RAG65.4171.8071.67
LightMem76.6478.6079.47
GAM74.7782.9885.79
A-MEM72.2176.0379.48
MemBox (Trace)78.4583.5375.58
AMA82.8882.6384.39
E-MEM83.6983.4881.38
S-MEM85.9587.4187.89

S-MEM leads A-MEM by +5.00, +6.20, and +4.00 percentage points under the three respective backbones. On LoCoMo, S-MEM also leads the strongest baseline in overall judge accuracy by 2.26, 3.88, and 2.10 percentage points, respectively. These results show that the observed improvements are not confined to lexical overlap under the controlled evaluation protocol.

We thank the reviewer again for the suggestions, which helped us clarify the contribution and strengthen the empirical evaluation.

Reviewer_58KiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how memory agents should be organized so that an LLM assistant can answer questions that require connecting evidence spread across sessions. Existing systems either store extracted fragments in external stores, which breaks the original context, or — like E-MEM — tie each agent's context to a temporal episode, which splits related experiences across agents. The authors propose S-MEM, which instead gives each memory agent a persistent topic: interactions are routed to topic-specific Specialists that accumulate all related history across sessions in their own context, while a Master Agent decides which Specialists to consult and combines their local findings into an answer.

The paper evaluates this design on LoCoMo and LongMemEval-S against a range of memory baselines, with ablations on local reasoning, assignment strategy, block size, and backbone choice. The headline claim is that topic organization improves long-term memory performance, especially on questions requiring cross-session synthesis, at the cost of substantially higher token consumption, which the authors disclose.

Critical strengths

The design question is real and the answer is sensible. Scattering related evidence from different sessions across episode-bound agents is a genuine failure mode of the memory-agent paradigm, and letting each topic accumulate its own history in one reasoning context is a clean, well-motivated fix. The gains concentrate exactly where this mechanism should help — multi-hop and temporal questions — and they largely disappear when Specialist local reasoning is removed, which is the strongest evidence in the paper that the mechanism does real work rather than riding on the backbone model.

The paper is unusually complete and honest for an agent-system submission. Prompts, inference settings, and costs are all disclosed, and the few results that cut against the story (adversarial questions favoring primary-only assignment, the open-domain regression) are reported rather than buried. I checked the main tables cell by cell, and the numbers match the text claims.

The ablations ask the right questions. Beyond the local-reasoning ablation, the multi-assignment, block-size, and role-wise backbone analyses each probe a place where the design could have broken, and the efficiency section is upfront about what the method costs.

Critical weaknesses

The core contribution is thin. S-MEM is E-MEM with the same memory agents re-partitioned by topic instead of episode, and topic-based memory organization is itself well established. The closest topical-memory work (Membox) is cited but never engaged with, and no topic-organized baseline is compared, so what is genuinely new here comes down mostly to prompt design.

The gains are not separated from the extra compute. S-MEM spends roughly three times the tokens of E-MEM per query and there is no budget-matched comparison, so it is unclear how much of the improvement comes from the organization itself rather than from simply doing more LLM work. On the harder benchmark the F1 margins over the best baseline are under one point, no significance testing is reported anywhere, and the paper's own backbone table shows the same configuration scoring differently across blocks — the apparent run noise may already be comparable to the effects being claimed.

The evaluation looks benchmark-bound. The prompts encode answering protocols tailored to LoCoMo and LongMemEval question formats, and the task-specific answer instructions that drive those protocols are never released. Baseline prompt effort is not disclosed either. This makes it hard to tell whether the paper is measuring memory organization or prompt tuning.

There are consistency problems that should not have survived proofreading. The quoted LoCoMo statistics do not match the official dataset paper, and the Master prompt asks the model to verify fields that the Specialists never output, which points to leftover artifacts from an earlier pipeline.

Questions for authors

Please see the weaknesses above.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the updated comments. We address the four concerns directly below.

1. Novelty is thin.(W1)

We respectfully disagree that S-MEM merely re-partitions E-MEM by topic. Our core contribution is agent specialization, not topic partitioning. In S-MEM, each Specialist takes responsibility for a specific topic, continuously accumulating and reasoning over its associated interactions across sessions. To enable this, we develop an integrated framework for Specialist registration, interaction allocation, query-driven activation, local reasoning, and cross-Specialist synthesis. This goes beyond reorganizing existing memory agents: it establishes a framework in which agents have persistent, specialized responsibilities for managing and reasoning over conversational history.

We now compare with Membox and AMA. Across three backbones on both benchmarks, S-MEM outperforms both; against the stronger baseline for each backbone, its overall LLM-as-judge accuracy improves by 3.07–3.88 points on LoCoMo and 8.80–17.00 points on LongMemEval-S.

LoCoMoGPT-4o-miniQwen3.5-27BGPT-6.1-terra
MethodF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACC
MemBox60.8054.8178.4565.5259.8983.5357.6152.6275.58
AMA63.8358.1182.8865.7360.8382.6368.9960.4084.39
S-MEM67.5760.0085.9569.6062.1987.4170.6262.6987.89
LongMemEval-SGPT-4o-miniQwen3.5-27BGPT-6.1-terra
MethodF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACC
MemBox43.3034.2962.4048.6038.9371.2050.1339.4179.00
AMA46.5836.9565.2051.2542.8375.2054.1543.9681.60
S-MEM57.5350.3982.2061.2957.5988.8063.4660.0790.40

A controlled within-S-MEM ablation replacing topic histories with episode histories lowers LoCoMo overall F1 from 67.39 to 63.10 and judge accuracy from 86.19% to 79.18%. These results support the value of topic-specialized agents; we do not treat the partitioning ablation as proof of local reasoning alone.

2. Compute and statistical uncertainty.(W2)

The gains are not explained by extra tokens alone. We increased E-MEM's query-time reasoning to closely match S-MEM's budget on LoCoMo:

MethodTokens/questionF1ACC
E-MEM (2 rounds × 8 agents)107,83563.5781.81%
E-MEM (3 rounds × 5 agents)110,01662.7480.64%
S-MEM112,58467.5785.95%

With token usage within 4.3%, S-MEM still leads the stronger E-MEM setting by 4.00 F1 and 4.14 LLM-as-judge accuracy points.

To assess statistical significance, we conducted paired bootstrap tests and report 95% confidence intervals against A-MEM, the strongest baseline on LongMemEval-S. S-MEM outperforms A-MEM by +0.88 F1 [0.10, 2.50] with GPT-4o-mini and +0.46 F1 [0.25, 2.40] with Qwen3.5-27B. Both confidence intervals exclude zero, supporting the statistical significance of these improvements.

Importantly, LLM-as-judge evaluation shows substantially larger gains: S-MEM achieves 82.2% vs. 77.2% with GPT-4o-mini and 88.8% vs. 82.6% with Qwen3.5-27B, outperforming A-MEM by 5.0 and 6.2 percentage points, respectively.

3. Prompt fairness and reproducibility.(W3)

We follow E-MEM’s evaluation prompting setup, rather than designing answer protocols specifically for S-MEM. Crucially, S-MEM and all compared methods use the same task-specific answer instructions on LoCoMo, ensuring that differences in answer-generation prompts do not favor S-MEM. These instructions were adapted from E-MEM’s publicly available evaluation implementation, and we now provide the full answer_instructions in Appendix F.1. Thus, the reported comparisons do not give S-MEM a prompt-specific advantage.

4. Reporting corrections.(W4)

We corrected the LoCoMo description to the official ACL 2024 release (10 conversations, 1,986 questions); the actual evaluation already used this full release, so results are unchanged. We also clarified the omitted code-level mapping that converts Specialist outputs into the four fields expected by the Master. The backbone-table discrepancy was a Conv-2 score mistakenly entered in a Conv-1 block; we corrected it to 66.41 F1 / 60.65 BLEU-1 and limit conclusions from this single-run analysis. We also removed “consistently” from the abstract, corrected the activation JSON example and query-time token accounting, and added the missing implementation details.

Thank you for the clarifications. I have no other questions, and I will maintain my original rating.

Reviewer_31qaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes S-MEM, a long-term memory framework that organizes conversational histories into persistent, topic-specific memory agents called Specialists. An Allocator assigns incoming interaction blocks to existing Specialists or creates new ones, preserving complete blocks across sessions. At query time, a Master activates relevant Specialists, which extract evidence and perform local reasoning over their accumulated histories; the Master then synthesizes their outputs and may request further information. Experiments on LoCoMo and LongMemEval-S compare S-MEM with existing memory systems, while ablations examine local reasoning, memory assignment, and backbone selection.

Critical strengths

The proposed agent architecture is in general intuitive and well-motivated: topic-specific Specialists accumulate related interactions across sessions, while a Master coordinates their activation and synthesizes their outputs.

The paper is generally well written, with a clear explanation of the motivation, architecture, and reasoning workflow.

Critical weaknesses
  1. The experiments do not sufficiently isolate the benefit of topic-specific Specialists over simpler topic-organized memory pipelines. Since specialization arises from retained histories rather than separately trained models, a baseline using topic-indexed raw conversation blocks and a shared LLM reader could preserve the same core principle. The local-reasoning ablation partly addresses this concern, but simultaneously removes evidence extraction and intermediate computation while shifting long-context reading to a different backbone. Consequently, the gains cannot be clearly attributed to the proposed Specialist architecture. Comparison with closely related topic-continuity methods such as Membox would further clarify the contribution.

  2. The generality of S-MEM’s gains across backbone capabilities remains insufficiently demonstrated. The main baseline comparisons use only GPT-4o-mini and Qwen3.5-27B as Master backbones. Although the backbone-sensitivity analysis includes DeepSeek-V4-Flash, it is restricted to one LoCoMo conversation and does not compare against baselines under that configuration. Consequently, it remains unclear whether S-MEM’s relative advantages persist with stronger reasoning and long-context models, which may benefit less from the proposed decomposition.

Questions for authors
  1. How does S-MEM compare with fixed topic buckets using the same local reader and synthesis prompts, without dynamic Specialist creation?

  2. Could the authors report comparisons against the main baselines using additional, stronger LLM backbones on both benchmarks?

  3. Given that S-MEM consumes more tokens than E-MEM, do its gains persist when both methods receive comparable token or inference-cost budgets?

  4. Does Table 10 include input, output, and reasoning tokens across all routing, Specialist, and Master calls? Are memory-construction costs excluded or amortized?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the constructive suggestions. We conducted additional controlled ablations, added comparisons with MemBox and a stronger Master backbone, and revised the token accounting. We address each concern below.

1. Isolating the contribution of topic-specific Specialists (W1, Q1)

Our contribution is not merely indexing conversation blocks by topic; it is the combination of persistent, dynamically formed topic contexts and query-time local reasoning before global synthesis. To clarify that, we add three complementary experiments, each addressing a distinct aspect of the concern.

(a) Dynamic versus fixed topic organization

With the reader and synthesis pipeline held constant, dynamic topic formation improves performance over fixed topic buckets. We construct the requested fixed-topic baseline by assigning the same raw conversation blocks to a predefined set of topics. For the fair comparison, both variants use the same LLM backbone, local reader, and reasoning/synthesis prompts. On LoCoMo, dynamic organization improves overall F1 from 67.39 to 68.24 and LLM-as-judge accuracy from 86.19% to 87.89% . The clearest category-level gains appear in multi-hop F1 (49.89 → 53.21) and temporal F1 (64.18 → 68.26).

VariantOverallMulti-HopTemporalOpen DomainSingle-HopAdversarial
F1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACC
S-MEM68.2460.4787.8953.2146.1170.4268.2654.7691.6732.6424.4071.7461.5351.6989.2397.1996.8197.47
Fixed Topics67.3959.6986.1949.8942.9268.3164.1850.9884.6230.5920.9869.5761.4551.4988.2896.3495.9796.62

(b) Matched-backbone ablation of Specialist local processing

The benefit of the Specialist processing stage persists after removing the backbone mismatch in our original ablation. We repeat the ablation on the first LoCoMo conversation using the same LLM backbone for every applicable role in both variants. In the ablation, Specialist local processing is removed and the Master reads the corresponding histories directly. Under this controlled comparison, S-MEM improves overall F1 from 58.02 to 66.61 and LLM-as-judge accuracy from 86.93% to 91.46%.

VariantOverallMulti-HopTemporalOpen DomainSingle-HopAdversarial
F1BLEU‑1Acc.F1BLEU‑1Acc.F1BLEU‑1Acc.F1BLEU‑1Acc.F1BLEU‑1Acc.F1BLEU‑1Acc.
S-MEM66.6159.3491.4649.2341.6093.7573.3468.6197.3037.1315.3692.3158.3547.5485.7193.6293.8593.62
w/o Local Reasoning58.0251.7186.9345.1239.7681.2571.2765.8994.5930.8112.9484.6251.5743.1478.5791.8791.8791.87

(c) Comparison with MemBox

We also add the requested MemBox (Trace) baseline on both benchmarks with all three backbones. S-MEM outperforms MemBox across all reported overall metrics and backbones. On LoCoMo, S-MEM improves overall F1 by 4.08–13.01 points and overall LLM-as-judge accuracy by 3.88–12.31 percentage points. On LongMemEval-S, the improvements are 12.69–14.23 F1 points and 11.40–19.80 points in LLM-as-judge accuracy.

LoCoMoGPT-4o-miniQwen3.5-27BGPT-6.1-terra
MethodF1BLEU-1Acc.F1BLEU-1Acc.F1BLEU-1Acc.
MemBox60.8054.8178.4565.5259.8983.5357.6152.6275.58
S-MEM67.5760.0085.9569.6062.1987.4170.6262.6987.89
LongMemEval-SGPT-4o-miniQwen3.5-27BGPT-6.1-terra
MethodF1BLEU-1Acc.F1BLEU-1Acc.F1BLEU-1Acc.
MemBox43.3034.2962.4048.6038.9371.2050.1339.4179.00
S-MEM57.5350.3982.2061.2957.5988.8063.4660.0790.40

Together, these experiments distinguish the benefits of adaptive topic formation, Specialist-stage local processing, and overall performance relative to existing topic-continuity memory.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

2. Generality under a stronger LLM backbone (W2, Q2)

S-MEM's relative advantage persists when the Master uses the stronger GPT-6.1-terra backbone, on both LoCoMo and LongMemEval-S. To address the limitation of our original single-conversation sensitivity analysis, we extend the full baseline comparisons to both benchmarks.

On LoCoMo, S-MEM achieves 70.62 F1 / 62.69 BLEU-1 / 87.89% LLM-as-judge accuracy and ranks first on 17 of 18 reported overall and category-level scores. The best competing overall LLM-as-judge accuracy is 85.79% (GAM), giving S-MEM a +2.10 percentage-point advantage. On LongMemEval-S, S-MEM scores 63.46 F1 / 60.07 BLEU-1 / 90.40% LLM-as-judge accuracy, exceeding the strongest baseline (A-MEM) by +1.85 F1, +4.28 BLEU-1, and +4.00 percentage points in LLM-as-judge accuracy.

LoCoMo — GPT-6.1-terra Master

BackboneMethodOverallMulti-HopTemporalOpen DomainSingle-HopAdversarial
F1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACCF1BLEU‑1ACC
GPT-6.1-terraFull Context68.4454.3883.7851.8536.3457.7564.8144.0687.8231.6920.5550.0063.0143.6586.0897.4797.4797.47
Naïve RAG59.5548.4971.6732.0120.4726.7656.2840.8373.7216.7612.5734.7853.5737.2875.8497.0597.0597.05
LightMem63.5860.2979.4741.7434.3054.9353.0347.9372.4416.3813.1030.4344.1538.7268.9093.3293.2993.25
GAM65.3160.3785.7950.6341.7276.5867.5362.1689.1023.7519.4250.0060.5554.2785.7689.1189.0889.03
A-MEM51.3545.2779.4834.7923.1477.1749.2138.3874.3613.9922.5163.0446.3939.9879.4378.6878.7186.92
MemBox57.6152.6275.5842.0533.9864.0864.9858.1685.9023.3318.5950.0060.6454.5585.4163.3863.3463.29
AMA68.9960.4084.3950.0841.4774.6564.8859.9188.4626.2720.5450.0062.4353.8683.0695.8595.8295.78
E-MEM64.6057.2881.3843.5633.0463.3860.2845.4076.9224.5318.2852.1761.7153.3485.8992.9092.9392.83
S-MEM70.6262.6987.8952.1143.0279.0172.8161.3292.3135.3029.5070.2264.9956.1687.0898.8398.6298.89

LongMemEval-S — GPT-6.1-terra Master

BackboneMethodF1BLEU-1ACC
GPT-6.1-terraFull Context51.0737.4380.40
Naïve RAG44.4132.1470.20
LightMem38.5431.2382.20
GAM46.5035.0776.40
A-MEM61.6155.7986.40
MemBox50.1339.4179.00
AMA54.1543.9681.60
E-MEM60.8651.6083.00
S-MEM63.4660.0790.40

Importantly, direct full-context inference also uses GPT-6.1-terra. Relative to Full Context, S-MEM improves overall LLM-as-judge accuracy from 83.78% to 87.89% (+4.11 points) on LoCoMo and from 80.40% to 90.40% (+10.00 points) on LongMemEval-S. Thus, the additional results indicate that stronger long-context capabilities do not eliminate the measured benefit of S-MEM's organization and hierarchical processing in these settings. We will add the complete comparisons to the revised paper.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

3. Performance at comparable token budgets (Q3)

Yes. S-MEM remains more accurate than E-MEM when their query-time token consumption is closely matched on LoCoMo. We increase E-MEM's inference budget from its original configuration (one reasoning round, up to five activated memory agents) in two ways: 3×5 permits up to three rounds with five agents per round, and 2×8 permits up to two rounds with eight agents per round. This tests whether giving E-MEM additional retrieval/reasoning opportunities closes the gap, rather than comparing only against its original lower-token setting.

ConfigurationTotal tokens / questionMaster tokens / questionMemory agent tokens / questionF1BLEU-1LLM-judge accuracy (%)
E-MEM 2×8107,834.918,284.2499,550.6763.5754.3881.81
E-MEM 3×5110,015.6410,792.8599,222.7962.7453.8380.64
S-MEM112,584.2210,988.02101,596.2067.5760.0085.95

4. Clarifying token accounting in Table 10 (Q4)

Table 10's original S-MEM total mixed query-time inference with two costs that should have been reported separately. We have corrected the accounting. All input, output, and reasoning tokens from query-time Master and Specialist calls are included, with query routing included in the Master total. The original S-MEM average additionally included the Allocator's offline memory-construction cost (amortized over the evaluated questions) and LLM-as-judge evaluation tokens, which are not part of query-time inference.

After excluding these two components, S-MEM's average changes from 113,388.60 to 112,584.22 tokens per question. The corrected query-time total consists of 10,988.02 Master tokens and 101,596.20 Specialist tokens per question. We separately report offline memory construction at 619.9 amortized tokens per question; the remaining 184.48 tokens per question in the prior total are attributable to judging under the stated accounting. The baseline token figures remain unchanged because their reported totals already excluded offline construction and judging. We apologize for the inconsistent accounting and will update Table 10, its caption, and the computational-cost discussion to distinguish query-time inference, offline memory construction, and evaluation overhead explicitly.

We thank the reviewer again for identifying these issues; the added matched comparisons and corrected cost breakdown will be included in the revised manuscript.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely thank you and the reviewers for your time and constructive feedback. We would like to briefly highlight several points that we believe are important for the final assessment of our submission.

First, regarding novelty, we acknowledge that topic-based memory organization is not new. However, the central contribution of S-MEM is not topic organization itself, but the use of specialized agents that maintain topic-specific interaction histories and perform local reasoning before global answer synthesis. This moves beyond organizing memory for retrieval toward organizing memory for reasoning.

Second, regarding the fairness of comparisons, we have conducted extensive additional experiments, including stronger LLM backbones, comparisons under comparable token budgets, and comparisons against topic-oriented memory approaches. These results provide further evidence that S-MEM's improvements cannot be explained solely by increased context access, token consumption, or topic-based organization.

Third, regarding empirical validation, we have provided additional ablations, expanded benchmark evaluations, and statistical analyses. We have also clarified the limitations of our results and revised claims where the evidence does not support stronger conclusions.

We recognize that S-MEM is built upon relatively simple components. Nevertheless, we believe its contribution lies in demonstrating that specializing memory agents by interaction topics, together with local reasoning and global synthesis, offers a meaningful and effective approach to long-term conversational memory. The additional experiments further support this distinction.

We respectfully hope that the AC will consider both the original submission and the substantial additional evidence provided during the rebuttal when making the final decision. We believe the work offers a valuable contribution to the design of long-term memory systems for LLM agents.

Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.