- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DyadMem, a dual-zone memory framework for long-horizon LLM agents. It combines Situation Memory, which maintains a cumulative summary of task states, with an Event List, which preserves chronological event summaries linked to original interaction records. The framework additionally supports hierarchical memory compression and source-grounded retrieval through natural-language search, event-ID lookup, and focused re-reading. Experiments on CAME-Bench and AMA-Bench demonstrate consistent improvements over long-context, retrieval-based, and memory-augmented baselines across two backbone models.
Critical strengths
-
Strong empirical results. DyadMem achieves 76.17% F1 on CAME-Bench and 67.75% accuracy on AMA-Bench with Gemma-4-31B, outperforming eight baselines. The improvements are reproduced with Qwen3-32B.
-
Reasonable motivation and practical design. Combining cumulative task-state summaries with chronological event indexing and raw-record retrieval addresses a genuine limitation of compressed agent memory.
-
Meaningful component ablations. Removing Situation Memory or Raw Recall reduces CAME-Bench performance by approximately 9 points, suggesting that both contribute to the framework.
Critical weaknesses
1. Limited methodological novelty.
The proposed framework primarily combines established ideas: cumulative summarization, chronological indexing, hierarchical compression, and source-grounded retrieval. Similar mechanisms have already been explored in ReadAgent, STITCH, AMA-Agent, and other agent-memory systems.
The paper does not clearly demonstrate a fundamentally new memory mechanism beyond their integration.
2. The ablations provide limited evidence for the novel design choices.
Table 3 is particularly concerning:
| Design component | F1 improvement |
|---|---|
| Hierarchical compression vs. FIFO | +1.09 |
| Explicit Event IDs | +1.34 |
| Named-anchor expansion and connection hints | +0.64 |
| Proposed retrieval vs. BM25 retrieval | +0.93 |
These improvements are relatively small. Moreover, removing Event IDs actually improves performance on the Large subset.
The substantial improvement from introducing Event List demonstrates the importance of retaining historical information, but does not establish that the specific proposed indexing design is superior to simpler, budget-matched alternatives.
3. Insufficient evidence of practical effectiveness and computational efficiency.
Although the paper targets long-horizon agent tasks, the evaluation primarily measures question-answering accuracy over recorded trajectories rather than improvements in actual agent execution.
Furthermore, DyadMem uses repeated memory construction, compression, drafting, and potentially multiple retrieval calls. The paper does not provide a systematic comparison of total token consumption, inference latency, or computational cost.
The baseline comparison also deserves closer scrutiny: the standalone RAG baselines do not use iterative retrieval, whereas DyadMem can perform multi-round recall.
Questions for authors
-
Methodological novelty: How does DyadMem fundamentally differ from existing memory architectures such as ReadAgent, STITCH, and AMA-Agent? Could the authors compare against a simpler baseline combining chronological summarization with raw-record retrieval under an identical memory budget?
-
Limited gains from the proposed design choices: Table 3 shows that hierarchical compression, explicit Event IDs, and the proposed retrieval refinements provide only modest improvements (approximately 0.6–1.3 F1 points). Moreover, removing Event IDs improves performance on the Large subset. How do the authors justify the necessity of these specific components? Are these improvements statistically significant?
-
Baseline fairness and computational efficiency: DyadMem benefits from multi-round retrieval and draft-then-verify reasoning, while standalone RAG baselines use single-pass retrieval. Could the authors evaluate stronger baselines equipped with comparable iterative retrieval capabilities? Additionally, how do total token consumption, inference latency, and computational costs compare when memory construction and repeated compression are included?
-
Generalization to real agent execution: The experiments evaluate memory-based question answering over recorded trajectories rather than actual long-horizon agent execution. Can the authors demonstrate that DyadMem improves downstream task success in interactive agent environments, rather than primarily improving retrospective information retrieval?
4 comments
Response to Reviewer 3 (Part 2/3)
W3 / Q3: Iterative-retrieval baseline
We thank the reviewer for this suggestion; this baseline should indeed be included. We added an Agentic Dense RAG baseline. It uses the same index as Dense RAG, but the model retrieves autonomously via tool calls for up to 6 rounds (vs. 3 for DyadMem) and can rewrite its queries; the context is likewise 32k.
| Method | CAME/Gemma F1 | CAME/Qwen F1 | AMA/Gemma Acc | AMA/Qwen Acc |
|---|---|---|---|---|
| Dense RAG | 36.93 | 33.92 | 46.47 | 50.00 |
| Agentic Dense RAG | 44.87 | 29.36 | 53.04 | 48.72 |
| DyadMem | 76.17 | 64.29 | 67.75 | 66.15 |
Iterative retrieval helps on Gemma and slightly hurts on Qwen, and the gap to DyadMem remains 14.7–34.9 points. In addition, the Simple control uses the same draft-then-verify procedure and reaches only 60.53 F1. Multi-round retrieval and verification by themselves therefore do not explain the main gains.
W3 / Q3: Computational cost
We measured the token cost of generation-model calls for all 10 methods, split into memory construction (including all re-compressions) and answering. Amortized cost = (total construction cost + total answering cost) ÷ total number of questions. Each cell reads "total input (of which non-cached input) / output"; total input includes server-side prefix-cache hits and excludes embedding and judge calls. "–" means the stage does not call the generation model.
CAME/Gemma (14 trajectories, 373 questions)
| Method | F1 | Construction (per trajectory) | Answering (per question) | Amortized (per question) |
|---|---|---|---|---|
| DyadMem | 76.17 | 480.0k (443.0k) / 34.7k | 26.6k (5.0k) / 0.08k | 44.6k (21.7k) / 1.4k |
| Long Context (256k) | 72.98 | – | 142.9k (45.3k) / 0.02k | 142.9k (45.3k) / 0.02k |
| AMA-Agent | 37.68 | 195.8k (192.9k) / 27.7k | 11.7k (9.9k) / 1.2k | 19.1k (17.2k) / 2.2k |
| Agentic Dense RAG | 44.87 | – | 33.2k (24.5k) / 0.25k | 33.2k (24.5k) / 0.25k |
| ReadAgent-P | 44.28 | 601.5k (553.2k) / 58.9k | 24.8k (5.7k) / 0.23k | 47.3k (26.4k) / 2.4k |
| STITCH | 44.93 | 15.2M (14.3M) / 114.3k | 23.6k (13.9k) / 1.2k | 594.2k (548.8k) / 5.5k |
| HippoRAG 2 | 29.39 | 353.2k (308.7k) / 18.5k | 7.9k (5.3k) / 0.21k | 21.1k (16.9k) / 0.90k |
| Dense RAG | 36.93 | – | 4.4k (4.2k) / 0.02k | 4.4k (4.2k) / 0.02k |
| BM25 RAG | 23.74 | – | 4.4k (4.2k) / 0.02k | 4.4k (4.2k) / 0.02k |
| MemGPT | 12.44 | – | 12.8k (9.3k) / 0.07k | 12.8k (9.3k) / 0.07k |
AMA/Gemma (208 trajectories, 2,496 questions)
| Method | Acc | Construction (per trajectory) | Answering (per question) | Amortized (per question) |
|---|---|---|---|---|
| DyadMem | 67.75 | 75.1k (74.5k) / 4.5k | 31.1k (21.2k) / 0.37k | 37.3k (27.4k) / 0.74k |
| Long Context (256k) | 60.94 | – | 54.1k (26.8k) / 0.09k | 54.1k (26.8k) / 0.09k |
| AMA-Agent | 60.22 | 50.3k (49.2k) / 6.4k | 9.1k (8.5k) / 0.35k | 13.3k (12.6k) / 0.88k |
| Agentic Dense RAG | 53.04 | – | 19.7k (12.5k) / 0.36k | 19.7k (12.5k) / 0.36k |
| ReadAgent-P | 49.84 | 408.5k (394.5k) / 22.4k | 14.0k (6.2k) / 0.12k | 48.0k (39.1k) / 2.0k |
| STITCH | 30.97 | 1.22M (1.06M) / 15.9k | 14.1k (9.1k) / 0.70k | 115.6k (97.5k) / 2.0k |
| HippoRAG 2 | 45.43 | 173.5k (153.3k) / 18.5k | 11.6k (9.3k) / 0.12k | 26.1k (22.0k) / 1.7k |
| Dense RAG | 46.47 | – | 4.5k (4.0k) / 0.09k | 4.5k (4.0k) / 0.09k |
| BM25 RAG | 41.03 | – | 4.5k (3.9k) / 0.09k | 4.5k (3.9k) / 0.09k |
| MemGPT | 30.21 | – | 18.3k (13.7k) / 0.15k | 18.3k (13.7k) / 0.15k |
Latency. Concurrency and caching conditions differ across methods in the full runs, so their speeds are not directly comparable. We therefore ran a controlled experiment: on AMA/Gemma, we fixed 18 trajectories and 216 questions (6 domains × one short/medium/long trajectory each), and every method built its memory from scratch and answered questions one by one on a single instance with a single concurrent request.
| Method | Construction (s/trajectory) | Answering (s/question) | Amortized (s/question) |
|---|---|---|---|
| DyadMem | 228.5 | 23.5 | 42.5 |
| Long Context (256k) | – | 73.9 | 73.9 |
| AMA-Agent | 236.0 | 20.7 | 40.3 |
| Agentic Dense RAG | 11.4 | 31.9 | 32.8 |
| ReadAgent-P | 1,444.3 | 9.6 | 130.0 |
| STITCH† | 1,327.2 | 43.2 | 136.0 |
| HippoRAG 2† | 1,045.4 | 13.4 | 98.3 |
| Dense RAG | 11.4 | 7.9 | 8.9 |
| BM25 RAG | 0.5 | 7.4 | 7.5 |
| MemGPT | 27.9 | 11.9 | 14.2 |
The construction time of Dense RAG and Agentic Dense RAG is the time to build the embedding index. † These two methods failed to complete some questions on this subset. The Answering column averages over completed questions, while the Amortized column is computed over all 216 questions with no answering time counted for unfinished ones; their amortized values are therefore underestimated.
Response to Reviewer 3 (Part 3/3)
W3 / Q3: Computational cost (continued)
These results show three things:
- Answering overhead is moderate. The constructed memory is shared across questions and reused as a cached prefix at answer time. On CAME, only 5.0k of DyadMem's 26.6k answering input per question is non-cached, comparable to Dense RAG (4.2k), and 57% of the questions invoke no retrieval at all. In the controlled experiment, DyadMem takes 23.5 s per question to answer, close to AMA-Agent (20.7 s).
- Construction is a one-time cost when multiple questions are asked on the same trajectory. Memory construction is question-independent, so the more questions per trajectory, the lower the per-question share. DyadMem's construction time is comparable to AMA-Agent's (228.5 s vs. 236.0 s per trajectory) and only about 1/6 to 1/5 of that of HippoRAG 2, STITCH, and ReadAgent-P.
- Lower input tokens and latency than methods of similar accuracy. The baseline closest in accuracy is Long Context. Compared with it, DyadMem uses less total input per question on both benchmarks (44.6k vs. 142.9k; 37.3k vs. 54.1k); its non-cached input is about half on CAME (21.7k vs. 45.3k) and roughly equal on AMA (27.4k vs. 26.8k); and its latency drops from 73.9 s to 42.5 s per question. Methods with lower total input per question than DyadMem (AMA-Agent, Agentic Dense RAG, HippoRAG 2, MemGPT, and the RAG methods) are cheaper, but at least 31 points less accurate on CAME and at least 7.5 points less accurate on AMA.
Full cost statistics, including Qwen, will be added to the appendix of the revision.
Q4: Real interactive execution
The reviewer rightly points out that gains in retrospective QA do not necessarily translate into gains in online execution. To test this, we ran all 35 tasks on the Shopping Admin site of WebArena-Lite [1]. Here, memory is no longer built from pre-recorded trajectories but is updated incrementally as the agent executes, and the agent decides each next action based on its current memory. All methods use Gemma as the base model, with the same actor prompt, browser tools, and a 40-step action limit; only the history-memory module is replaced. Semantic judging uses Gemma. This setting places real demands on memory: 29 of DyadMem's 35 trajectories exceed 32k tokens, with a median trajectory length of about 116k tokens.
| Method | Success rate | Avg. steps |
|---|---|---|
| DyadMem | 17/35 (48.6%) | 23.8 |
| AMA-Agent | 14/35 (40.0%) | 25.0 |
| Agentic Dense RAG | 14/35 (40.0%) | 21.6 |
| STITCH | 13/35 (37.1%) | 25.3 |
| BM25 RAG | 12/35 (34.3%) | 26.5 |
| ReadAgent-P | 12/35 (34.3%) | 24.1 |
| Dense RAG | 11/35 (31.4%) | 25.3 |
| HippoRAG 2 | 11/35 (31.4%) | 26.2 |
| Long Context (256k) | 10/35 (28.6%) | 21.5 |
| MemGPT | 10/35 (28.6%) | 22.5 |
In this small-scale preliminary experiment, DyadMem achieves the highest success rate among the 10 settings, providing initial evidence that it can also serve as the history memory of an online agent. Given the limited number of tasks, differences of a few tasks between methods require larger-scale evaluation to confirm, which we leave to future work. We will add this experiment to the appendix.
We thank the reviewer again. The controls, significance tests, cost statistics, and online experiments added following these suggestions have made the paper's conclusions more solid and helped us delineate the role of each component more precisely. We would be glad to address any remaining questions during the discussion period, and if these additions resolve your main concerns, we would kindly ask you to consider revisiting your score.
References
[1] Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Xixuan Song, Yifan Xu, Shudan Zhang, Hanyu Lai, Jiadai Sun, Xinyue Yang, et al. VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents. In International Conference on Learning Representations (ICLR), 2025.
I appreciate the authors' efforts and find the additional evidence convincing. I have therefore decided to raise my score.
Response to Reviewer 3 (Part 1/3)
We thank the reviewer for recognizing our empirical results, motivation, and ablation design. We have completed all the suggested experiments and respond point by point below.
W1 / Q1: Methodological novelty
We agree that cumulative summarization, chronological indexing, hierarchical compression, and raw-record retrieval each have precedents on their own. Our contribution is a dual-zone principle for organizing memory:
Compared with existing methods, the main difference lies in how memory is organized. ReadAgent generates a gist once per page and never compresses it further, so gist memory grows linearly with the trajectory and exceeds the context in our 32k-budget experiments. STITCH organizes memory around contextual intents, and AMA-Agent builds a causal graph combined with tool-based retrieval; both organize history by semantic relations. DyadMem instead organizes history by time: alongside a separately maintained current state, it keeps a budget-bounded event index that covers the entire history and links to the raw records.
We have completed the control the reviewer suggested, "chronological summarization + raw-record retrieval under an identical memory budget" (Simple; see the General Response for the setup). Simple is such a simpler alternative: a single summary for both state and history, plus a coarse-grained table of contents.
DyadMem outperforms Simple by 15.64 F1, and by 25.08 and 16.88 on Medium and Large trajectories, respectively. This suggests that, under the same raw retrieval and memory budget, the combination of chronological summarization and raw retrieval cannot substitute for DyadMem's memory organization. Combined with the w/o Situation ablation (which keeps the per-record Event List and retrieval settings unchanged and removes only the state zone, reducing F1 by 9.01), this indicates that the state zone and the per-record history zone each contribute.
W2 / Q2: Necessity and statistical significance of the smaller design choices
Following the reviewer's suggestion, we ran a trajectory-level paired bootstrap (20,000 resamples) for each comparison in Table 3:
The first three rows correspond to the two memory zones and raw recall, and none of their intervals include zero; the Simple control above also supports the role of the dual-zone organization. The intervals of the last four rows include zero, and the reviewer's caution is reasonable. However, these four components play different roles and are best read separately:
We will add the confidence intervals to Table 3 in the revision.