- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether memory systems genuinely help coding agents over evolving software repositories. To study this question, the authors construct MemDev-Bench, which pairs 555 base tasks from SWE-bench Verified and Pro with 4,869 earlier PRs from the same repositories that are annotated for relevance (Strong Related, Weak Related, or Unrelated) and converted into executable tasks. Using the SWE-bench Verified subset under an information-isolation protocol, the authors evaluate five representative memory systems (along with 29 additional systems in the appendix) against a no-memory baseline and raw historical trajectories across three history scenarios. Their empirical analysis shows that while memory helps agents localize and edit lines on hard tasks, it also causes regressions on tasks the agent could already solve on its own, and compressed memories lose useful problem-solving process details compared to raw trajectories.
Critical strengths
The paper studies a very important and timely question for coding agents. As more and more memory systems emerge, the community still lacks a reliable way to measure whether they genuinely help or are mostly hype. This paper tackles the question head-on and presents balanced, honest findings rather than one-sided positive claims: while memory helps solve some hard tasks, it also misleads the agent into failing tasks that it could originally solve on its own. These findings offer practical insights for both designing and using agent memory systems.
Moreover, the benchmark is rich in scale and constructed with careful quality control. MemDev-Bench pairs 555 base tasks with 4,869 executable historical tasks across multiple repositories, and the authors put substantial effort into ensuring data quality.
Critical weaknesses
I have two major concerns about the benchmark design and experimental setup.
First, the constructed task histories in MemDev-Bench are not natural development streams, but are selected and pre-filtered around each target SWE task. In real-world software development, repository evolution is highly interleaved across diverse features and bug fixes, and the useful memory for a target task is often buried within a long, messy history of many unrelated PRs and commits. A core challenge for memory systems is therefore to identify relevant experience from massive real-world noise and manage memory over long-term accumulation. However, MemDev-Bench does not reflect this realistic setting, so claiming to evaluate over real interleaved development and long-term repository evolution feels like an overclaim. In the benchmark, each history is capped at only 35 tasks (L277) and is pre-filtered around the target task using reference links or high file-overlap and embedding similarity (> 0.65), even for tasks labeled as "Unrelated". In addition, because the code-history route requires sharing modified files with the target PR, useful cross-file histories are omitted unless developers explicitly linked them. According to the plastic surgery hypothesis in software engineering, the ingredients of a fix often already exist elsewhere in the codebase, not necessarily in the files being changed, so this rule likely misses important cross-file memories.
Another key issue is that the information-isolation protocol blocks all Git history commands across all conditions, making the NoMem baseline weaker than a real coding agent. Native Git commands such as git log, git show, and git blame are completely disabled in the baseline. In practice, however, coding agents often use these basic Git commands to inspect past changes, and these commands alone might be enough to solve many tasks without an external memory system. Preventing answer leakage only requires removing the Git history after the target task, so there is no need to block past history as well.
Overall, evaluating memory systems for coding agents is a timely and practical research topic, and I am happy to raise my score if the authors can address my concerns by: (1) running a small-scale validation on a few repositories using an unfiltered, continuous sequence of PRs before the target task to check whether the main findings still hold; (2) analyzing whether the current file-overlap filter misses critical cross-file memories and discussing possible ways to mitigate this issue; and (3) adding a comparison that allows agents to use past Git history, including at least a "NoMem + Git history" baseline.
Questions for authors
- Could the authors run a small-scale validation on a few repositories using an unfiltered, continuous sequence of PRs before the target task?
- Could the authors analyze to what extent the current file-overlap filter in the code-history route misses critical cross-file memories?
- How do the evaluated memory systems perform when compared against a baseline that allows agents to use native Git history commands (e.g., git log, git show, and git blame) on the repository history prior to the target task?
11 comments
Response to Q1 (2/3): Validation on unfiltered consecutive PR sequences
2. New experiment: validating the main findings on unfiltered PR sequences (Weakness 1: pre-selected histories)
2.1 Setup
- Tasks: 40 base tasks from 10 repositories, with 20 Improvement Tasks and 20 Preservation Tasks.
- History: For each base task, we take the 30 PRs merged most recently before its base commit. These PRs are converted into executable historical tasks and ordered by merge time, from earliest to latest. No relevance filtering is applied, and only temporal-validity and answer-leakage checks are performed.
- Evaluation: NoMem, FullTraj, A-Mem, and Claude-Mem, each with three independent runs.
Tables S2 and S3 report the results on the two subsets.
Table S2: Results on the 20 Improvement Tasks
| System | Resolved (%) | Rescue (%) | Harm (%) | Tok. In (M) | Tok. Out (K) | Steps |
|---|---|---|---|---|---|---|
| NoMem | 53.3 ± 5.8 | — | — | 5.42 ± 0.15 | 56.0 ± 4.0 | 150.4 ± 0.4 |
| FullTraj | 51.7 ± 7.6 | 12.8 [3.9, 25.0] | 14.4 [4.4, 26.7] | 37.09 ± 2.38 | 42.1 ± 2.6 | 108.7 ± 6.5 |
| A-Mem | 50.0 ± 0.0 | 7.8 [1.1, 16.7] | 11.1 [3.9, 19.4] | 5.79 ± 0.20 | 61.5 ± 2.8 | 159.9 ± 4.2 |
| Claude-Mem | 56.7 ± 7.6 | 11.1 [3.3, 20.0] | 7.8 [2.8, 13.3] | 5.30 ± 0.22 | 56.1 ± 2.1 | 151.0 ± 5.1 |
Table S3: Results on the 20 Preservation Tasks
| System | Resolved (%) | Rescue (%) | Harm (%) | Tok. In (M) | Tok. Out (K) | Steps |
|---|---|---|---|---|---|---|
| NoMem | 95.0 ± 0.0 | — | — | 2.83 ± 0.26 | 32.1 ± 0.4 | 110.0 ± 5.9 |
| FullTraj | 88.3 ± 7.6 | 3.3 [0.0, 7.8] | 10.0 [2.8, 19.4] | 21.73 ± 2.29 | 18.5 ± 3.2 | 67.9 ± 7.2 |
| A-Mem | 98.3 ± 2.9 | 5.0 [0.0, 10.0] | 1.7 [0.0, 5.0] | 2.84 ± 0.24 | 34.7 ± 2.0 | 110.0 ± 7.6 |
| Claude-Mem | 98.3 ± 2.9 | 4.4 [0.0, 9.4] | 1.1 [0.0, 3.3] | 2.71 ± 0.30 | 31.3 ± 1.5 | 106.8 ± 7.5 |
Note: Resolved, tokens, and steps report mean ± SD over three runs. Brackets for Rescue and Harm give 95% bootstrap CIs.
2.2 Findings consistent with the paper
The results reproduce three main findings of the paper. Given the small sample, we mainly compare the direction of each finding.
- FullTraj substantially shortens the solving process (§4.4). Pooled over both subsets, FullTraj uses 32.2% fewer steps than NoMem, close to about 34% in the paper.
- Regressions offset gains (§4.1). On Improvement Tasks, Rescue ranges from 7.8% to 12.8%, while Harm ranges from 7.8% to 14.4%.
- FullTraj is more sensitive to unrelated history (Table 3). On Preservation Tasks, its Resolved drops from 95.0% to 88.3%. This matches the direction of the drop under Noisy and Repo history in the paper.
Response to Q1 (3/3): New observations from natural development sequences
3. New observations from natural development sequences
Beyond the consistent findings above, the natural development sequences reveal three new observations.
(1) Useful experience is sparse, so memory gains are small. On Improvement Tasks, all three memory conditions stay close to NoMem in Resolved. To understand why, we labeled the 1,200 historical tasks with the annotation pipeline of the paper. Table S4 shows their relevance composition.
Table S4: Relevance composition of the unfiltered histories
| Group | Strong Related | Weak Related | Unrelated | Total |
|---|---|---|---|---|
| Improvement (20 base tasks) | 19 (3.17%) | 19 (3.17%) | 562 (93.67%) | 600 |
| Preservation (20 base tasks) | 16 (2.67%) | 17 (2.83%) | 567 (94.50%) | 600 |
| Total (40 base tasks) | 35 (2.92%) | 36 (3.00%) | 1,129 (94.08%) | 1,200 |
Related tasks make up only 5.9% of these natural sequences, compared with 40.1% in the Verified subset of MemDev-Bench (Table 1). We believe this sparsity is the main reason for the small gains. With little usable experience in the history, memory can offer only limited help.
(2) Memory systems are more robust than FullTraj to noise in natural history. On Preservation Tasks, A-Mem and Claude-Mem have a Harm of only 1.7% and 1.1%, whereas FullTraj reaches 10.0%. When most history is unrelated to the current task, compressing and selecting history reduces the regressions caused by irrelevant information.
(3) FullTraj's efficiency gain does not depend on relevance. Although 94.1% of the history is unrelated to the base task, FullTraj still uses 32.2% fewer steps than NoMem. Its step savings may therefore come largely from repository-level process information, not only from task-specific experience.
These results confirm your point that useful experience in real development is buried among many unrelated tasks. They also show why relevance-controlled histories are necessary. Only when a history contains enough usable experience can an evaluation assess whether a memory system exploits that experience effectively.
Revisions to the paper
We have added the history-length statistics, the new experiment, and the relevance labels to Appendix L of the revised paper (L2311–L2378, Tables 19–21). Section 3 now points to this appendix (L281–L282). All changes are marked in blue.
Thank you again for this valuable suggestion. The experiment makes our evaluation more complete and gives us a clearer view of the scope of the benchmark.
Response to Q1 (1/3): History length in MemDev-Bench
Thank you very much for this insightful suggestion. In Weakness 1, you raised two points. First, each history contains at most 35 tasks (L279–L280). Second, historical tasks are pre-selected around the target task, so they do not reflect real development, where useful experience is buried among many unrelated PRs. This point goes to the core of evaluating memory systems. This response addresses both points and Question 1. Part 1 explains the 35-task cap, and Parts 2 and 3 report and analyze a new experiment on unfiltered consecutive PR sequences, as you suggested. Your concern that the file-overlap rule may miss cross-file history is addressed in our response to Question 2.
1. History length (Weakness 1: the 35-task cap)
The 35-task cap is an evaluation setting, not a limit of the benchmark. As stated in §3 (L279–L281), the cap controls evaluation cost and avoids exceeding the context window. FullTraj places all raw historical trajectories into the agent's context. To keep the measured effect of memory undistorted, we also disable automatic context compaction (§3). Longer histories therefore make FullTraj much more expensive. Under Repo History, its input per task already reaches 5.1× that of NoMem (32.86M vs. 6.45M tokens).
MemDev-Bench itself supports much longer histories. Table S1 counts, for each base task, the executable historical tasks completed in the same repository before its base PR was created. In 15 of the 22 repositories, the median base task has more than 35 earlier historical tasks.
Table S1: Executable historical tasks available before each base task
Verified subset
Pro subset
As you pointed out, however, these histories are still mined around each base task, rather than taken from natural, unfiltered development sequences. Following your suggestion, we therefore ran the experiment in the next response.