acceptodds
Under review as a conference paper at ICLR 2027

MemDev-Bench: Does Memory Help Code Agents? Benchmarking Memory Systems over Real Repository Histories

Abstract

Code agents increasingly work on evolving real repositories, and memory systems aim to help them reuse earlier experience. However, such experience is diverse and evolving, so it remains unclear how memory systems organize it and thereby affect code agents. Existing benchmarks cannot answer this reliably because they lack either temporally valid histories or control over prior-task relevance. In this paper, we present MemDev-Bench to evaluate how memory systems affect code agents over diverse repository histories. It pairs each task with relevance-annotated earlier tasks from the same repository to build histories with controlled relevance. We also build an automated pipeline that reliably makes these earlier tasks executable and verifiable, producing 4,869 historical tasks for 555 base tasks. Based on MemDev-Bench, our systematic evaluation of memory systems across diverse development scenarios yields three main findings. (1) Memory mainly improves fine-grained localization and editing, increasing functional correctness. (2) Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures. These failures mainly stem from agents reusing memory without reassessing its applicability. (3) Compared with using memory systems, directly providing agents with raw historical trajectories substantially reduces their steps. Further analysis suggests that agents save these steps by reusing process information that compressed memories tend to omit. These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context.

Primary area:
datasets and benchmarks
Keywords:
software engineering agents, agent memory, code agents, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CjaNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether memory systems genuinely help coding agents over evolving software repositories. To study this question, the authors construct MemDev-Bench, which pairs 555 base tasks from SWE-bench Verified and Pro with 4,869 earlier PRs from the same repositories that are annotated for relevance (Strong Related, Weak Related, or Unrelated) and converted into executable tasks. Using the SWE-bench Verified subset under an information-isolation protocol, the authors evaluate five representative memory systems (along with 29 additional systems in the appendix) against a no-memory baseline and raw historical trajectories across three history scenarios. Their empirical analysis shows that while memory helps agents localize and edit lines on hard tasks, it also causes regressions on tasks the agent could already solve on its own, and compressed memories lose useful problem-solving process details compared to raw trajectories.

Critical strengths

The paper studies a very important and timely question for coding agents. As more and more memory systems emerge, the community still lacks a reliable way to measure whether they genuinely help or are mostly hype. This paper tackles the question head-on and presents balanced, honest findings rather than one-sided positive claims: while memory helps solve some hard tasks, it also misleads the agent into failing tasks that it could originally solve on its own. These findings offer practical insights for both designing and using agent memory systems.

Moreover, the benchmark is rich in scale and constructed with careful quality control. MemDev-Bench pairs 555 base tasks with 4,869 executable historical tasks across multiple repositories, and the authors put substantial effort into ensuring data quality.

Critical weaknesses

I have two major concerns about the benchmark design and experimental setup.

First, the constructed task histories in MemDev-Bench are not natural development streams, but are selected and pre-filtered around each target SWE task. In real-world software development, repository evolution is highly interleaved across diverse features and bug fixes, and the useful memory for a target task is often buried within a long, messy history of many unrelated PRs and commits. A core challenge for memory systems is therefore to identify relevant experience from massive real-world noise and manage memory over long-term accumulation. However, MemDev-Bench does not reflect this realistic setting, so claiming to evaluate over real interleaved development and long-term repository evolution feels like an overclaim. In the benchmark, each history is capped at only 35 tasks (L277) and is pre-filtered around the target task using reference links or high file-overlap and embedding similarity (> 0.65), even for tasks labeled as "Unrelated". In addition, because the code-history route requires sharing modified files with the target PR, useful cross-file histories are omitted unless developers explicitly linked them. According to the plastic surgery hypothesis in software engineering, the ingredients of a fix often already exist elsewhere in the codebase, not necessarily in the files being changed, so this rule likely misses important cross-file memories.

Another key issue is that the information-isolation protocol blocks all Git history commands across all conditions, making the NoMem baseline weaker than a real coding agent. Native Git commands such as git log, git show, and git blame are completely disabled in the baseline. In practice, however, coding agents often use these basic Git commands to inspect past changes, and these commands alone might be enough to solve many tasks without an external memory system. Preventing answer leakage only requires removing the Git history after the target task, so there is no need to block past history as well.

Overall, evaluating memory systems for coding agents is a timely and practical research topic, and I am happy to raise my score if the authors can address my concerns by: (1) running a small-scale validation on a few repositories using an unfiltered, continuous sequence of PRs before the target task to check whether the main findings still hold; (2) analyzing whether the current file-overlap filter misses critical cross-file memories and discussing possible ways to mitigate this issue; and (3) adding a comparison that allows agents to use past Git history, including at least a "NoMem + Git history" baseline.

Questions for authors
  1. Could the authors run a small-scale validation on a few repositories using an unfiltered, continuous sequence of PRs before the target task?
  2. Could the authors analyze to what extent the current file-overlap filter in the code-history route misses critical cross-file memories?
  3. How do the evaluated memory systems perform when compared against a baseline that allows agents to use native Git history commands (e.g., git log, git show, and git blame) on the repository history prior to the target task?
11 comments

Response to Q1 (1/3): History length in MemDev-Bench

Thank you very much for this insightful suggestion. In Weakness 1, you raised two points. First, each history contains at most 35 tasks (L279–L280). Second, historical tasks are pre-selected around the target task, so they do not reflect real development, where useful experience is buried among many unrelated PRs. This point goes to the core of evaluating memory systems. This response addresses both points and Question 1. Part 1 explains the 35-task cap, and Parts 2 and 3 report and analyze a new experiment on unfiltered consecutive PR sequences, as you suggested. Your concern that the file-overlap rule may miss cross-file history is addressed in our response to Question 2.

1. History length (Weakness 1: the 35-task cap)

The 35-task cap is an evaluation setting, not a limit of the benchmark. As stated in §3 (L279–L281), the cap controls evaluation cost and avoids exceeding the context window. FullTraj places all raw historical trajectories into the agent's context. To keep the measured effect of memory undistorted, we also disable automatic context compaction (§3). Longer histories therefore make FullTraj much more expensive. Under Repo History, its input per task already reaches 5.1× that of NoMem (32.86M vs. 6.45M tokens).

MemDev-Bench itself supports much longer histories. Table S1 counts, for each base task, the executable historical tasks completed in the same repository before its base PR was created. In 15 of the 22 repositories, the median base task has more than 35 earlier historical tasks.

Table S1: Executable historical tasks available before each base task

Verified subset

RepositoryBase tasksEarlier historical tasks per base task (min / median / max)
Django169126 / 727 / 893
Matplotlib3069 / 201.5 / 237
SymPy2961 / 131 / 166
Scikit-learn2792 / 162 / 202
Sphinx2269 / 125.5 / 176
Xarray1852 / 127.5 / 160
Astropy1648 / 118 / 126
Pytest1646 / 85 / 109
Requests821 / 45 / 66
Pylint616 / 54.5 / 61
Seaborn217 / 17.5 / 18
Flask113 / 13 / 13

Pro subset

RepositoryBase tasksEarlier historical tasks per base task (min / median / max)
OpenLibrary5040 / 178.5 / 250
Flipt4816 / 250.5 / 377
Vuls4398 / 191 / 263
Ansible2359 / 120 / 146
Teleport2311 / 67 / 105
Navidrome915 / 27 / 37
NodeBB517 / 22 / 29
Tutanota414 / 27.5 / 30
Element Web313 / 16 / 19
qutebrowser318 / 25 / 34

As you pointed out, however, these histories are still mined around each base task, rather than taken from natural, unfiltered development sequences. Following your suggestion, we therefore ran the experiment in the next response.

Response to Q1 (2/3): Validation on unfiltered consecutive PR sequences

2. New experiment: validating the main findings on unfiltered PR sequences (Weakness 1: pre-selected histories)

2.1 Setup

  • Tasks: 40 base tasks from 10 repositories, with 20 Improvement Tasks and 20 Preservation Tasks.
  • History: For each base task, we take the 30 PRs merged most recently before its base commit. These PRs are converted into executable historical tasks and ordered by merge time, from earliest to latest. No relevance filtering is applied, and only temporal-validity and answer-leakage checks are performed.
  • Evaluation: NoMem, FullTraj, A-Mem, and Claude-Mem, each with three independent runs.

Tables S2 and S3 report the results on the two subsets.

Table S2: Results on the 20 Improvement Tasks

SystemResolved (%)Rescue (%)Harm (%)Tok. In (M)Tok. Out (K)Steps
NoMem53.3 ± 5.8——5.42 ± 0.1556.0 ± 4.0150.4 ± 0.4
FullTraj51.7 ± 7.612.8 [3.9, 25.0]14.4 [4.4, 26.7]37.09 ± 2.3842.1 ± 2.6108.7 ± 6.5
A-Mem50.0 ± 0.07.8 [1.1, 16.7]11.1 [3.9, 19.4]5.79 ± 0.2061.5 ± 2.8159.9 ± 4.2
Claude-Mem56.7 ± 7.611.1 [3.3, 20.0]7.8 [2.8, 13.3]5.30 ± 0.2256.1 ± 2.1151.0 ± 5.1

Table S3: Results on the 20 Preservation Tasks

SystemResolved (%)Rescue (%)Harm (%)Tok. In (M)Tok. Out (K)Steps
NoMem95.0 ± 0.0——2.83 ± 0.2632.1 ± 0.4110.0 ± 5.9
FullTraj88.3 ± 7.63.3 [0.0, 7.8]10.0 [2.8, 19.4]21.73 ± 2.2918.5 ± 3.267.9 ± 7.2
A-Mem98.3 ± 2.95.0 [0.0, 10.0]1.7 [0.0, 5.0]2.84 ± 0.2434.7 ± 2.0110.0 ± 7.6
Claude-Mem98.3 ± 2.94.4 [0.0, 9.4]1.1 [0.0, 3.3]2.71 ± 0.3031.3 ± 1.5106.8 ± 7.5

Note: Resolved, tokens, and steps report mean ± SD over three runs. Brackets for Rescue and Harm give 95% bootstrap CIs.

2.2 Findings consistent with the paper

The results reproduce three main findings of the paper. Given the small sample, we mainly compare the direction of each finding.

  • FullTraj substantially shortens the solving process (§4.4). Pooled over both subsets, FullTraj uses 32.2% fewer steps than NoMem, close to about 34% in the paper.
  • Regressions offset gains (§4.1). On Improvement Tasks, Rescue ranges from 7.8% to 12.8%, while Harm ranges from 7.8% to 14.4%.
  • FullTraj is more sensitive to unrelated history (Table 3). On Preservation Tasks, its Resolved drops from 95.0% to 88.3%. This matches the direction of the drop under Noisy and Repo history in the paper.

Response to Q1 (3/3): New observations from natural development sequences

3. New observations from natural development sequences

Beyond the consistent findings above, the natural development sequences reveal three new observations.

(1) Useful experience is sparse, so memory gains are small. On Improvement Tasks, all three memory conditions stay close to NoMem in Resolved. To understand why, we labeled the 1,200 historical tasks with the annotation pipeline of the paper. Table S4 shows their relevance composition.

Table S4: Relevance composition of the unfiltered histories

GroupStrong RelatedWeak RelatedUnrelatedTotal
Improvement (20 base tasks)19 (3.17%)19 (3.17%)562 (93.67%)600
Preservation (20 base tasks)16 (2.67%)17 (2.83%)567 (94.50%)600
Total (40 base tasks)35 (2.92%)36 (3.00%)1,129 (94.08%)1,200

Related tasks make up only 5.9% of these natural sequences, compared with 40.1% in the Verified subset of MemDev-Bench (Table 1). We believe this sparsity is the main reason for the small gains. With little usable experience in the history, memory can offer only limited help.

(2) Memory systems are more robust than FullTraj to noise in natural history. On Preservation Tasks, A-Mem and Claude-Mem have a Harm of only 1.7% and 1.1%, whereas FullTraj reaches 10.0%. When most history is unrelated to the current task, compressing and selecting history reduces the regressions caused by irrelevant information.

(3) FullTraj's efficiency gain does not depend on relevance. Although 94.1% of the history is unrelated to the base task, FullTraj still uses 32.2% fewer steps than NoMem. Its step savings may therefore come largely from repository-level process information, not only from task-specific experience.

These results confirm your point that useful experience in real development is buried among many unrelated tasks. They also show why relevance-controlled histories are necessary. Only when a history contains enough usable experience can an evaluation assess whether a memory system exploits that experience effectively.

Revisions to the paper

We have added the history-length statistics, the new experiment, and the relevance labels to Appendix L of the revised paper (L2311–L2378, Tables 19–21). Section 3 now points to this appendix (L281–L282). All changes are marked in blue.

Thank you again for this valuable suggestion. The experiment makes our evaluation more complete and gives us a clearer view of the scope of the benchmark.

Reviewer_kDWJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemDev-Bench to evaluate memory systems for code agents using historical development tasks from real repositories. The benchmark links 555 base tasks with 4,869 executable historical tasks, checks temporal precedence and direct solution leakage, and annotates relevance to the current task. Experiments show that memory can both help and mislead agents; full historical trajectories reduce interaction steps but increase input tokens.

Critical strengths
  1. The benchmark combines executable repository histories, strict temporal precedence, and human-verified graded relevance, adding information beyond histories organized only by time or code overlap. Three additional engineers independently blind-label 2,270 samples, with Fleiss' kappa of 0.88 for relevance level and 0.80 for relation type, providing substantial support for the task-relation labels.

  2. The paper goes beyond success-rate rankings by examining consistently failed and consistently solved tasks and analyzing memory-influenced trajectories. These analyses show both beneficial reuse and misleading reuse; the reduction in steps but increase in input tokens with full historical trajectories also reveals a practical cost tradeoff.

Critical weaknesses
  1. The relevance labels and the material used to construct memory are different objects. Appendix C labels historical PR descriptions and fixes, whereas Section 3 constructs memory from agent-generated historical trajectories; Appendix E.1 explicitly retains FullTraj trajectories regardless of task outcome. The analysis does not verify that each trajectory contains the reusable information underlying its relevance label, so a failure to benefit from related history cannot cleanly distinguish ineffective memory use from missing useful experience.

  2. Noisy History adds unrelated tasks, changing both relevance and history length. It models interleaved development, but the increased Harm reported in Section 4.1 cannot distinguish interference from unrelated content from the burden of a longer history, leaving the source of failures insufficiently explained. The increase is consistent in direction across the five memory systems on Improvement Tasks but modest in size, so a length-matched comparison would help attribute it.

Questions for authors

None.

7 comments

Response to W1 (1/3): Why we label tasks, and evidence from memory gains

Thank you for the careful review and the very positive assessment! Your questions are insightful and in-depth, and they have been very helpful for improving the paper. We respond to each of them below.

We appreciate this sharp observation. It touches on a core issue in designing memory benchmarks and has helped us make the analysis more complete. Our response has three parts: (1) why we label tasks rather than trajectories; (2) evidence that the historical trajectories in our experiments do contain the information behind the labels; and (3) a new control experiment in which the memory material exactly matches the labeled content, showing that our conclusions are unaffected.

1. Design rationale: labeling tasks, not trajectories

  • Simulating real development. MemDev-Bench simulates how a code agent accumulates experience in real development. In practice, no one tells the agent whether each attempt is correct, so its memory can only come from its own trajectories. For this reason, FullTraj does not filter trajectories by outcome (Appendix E.1, L1443–L1444). Filtering or labeling trajectories by correctness would give memory systems oracle information that real development does not provide.
  • Trajectories cannot be labeled in advance. Trajectories do not exist when the benchmark is built. They are generated during evaluation by the agent and memory system under test, and they change with each agent and each run. Task labels, in contrast, are fixed. They ensure that all systems face the same histories, which keeps the comparison fair.

2. Historical trajectories do contain the information behind the labels

2.1 Indirect evidence: gains on tasks that NoMem consistently fails (§4.2, Table 5)

These 68 Improvement Tasks failed at least twice in the three screening runs, and then failed in all three independent NoMem runs. In total, each failed in at least 5 of 6 runs without memory. Gains on these tasks can therefore be attributed to memory with more confidence. Table R1 summarizes the 18 memory settings (FullTraj and five memory systems, each under three histories).

Table R1: Results on the 68 tasks that NoMem consistently fails (%)

MetricNoMem18 memory settingsSettings above NoMem
Resolved07.8–13.718/18
F2P18.123.2–28.618/18
File Recall74.270.4–76.82/18
Line Recall18.417.7–22.513/18
Edit Recall11.09.7–14.816/18

These results lead to two conclusions:

  • The gains come from memory and are consistent in direction. All conditions share the same agent configuration and isolation protocol. The only difference is access to memory built from historical trajectories. On these tasks, all 18 memory settings additionally solve 7.8%–13.7% of the tasks, and all of them raise F2P by 5.1–10.5 points. The historical trajectories therefore contain information that helps solve the tasks.
  • Memory tells the agent where and how to edit, which is the basis of the labels. Without memory, the agent already covers 74.2% of the gold files, and memory barely changes this. The gains concentrate on Line Recall and Edit Recall within files. In Appendix C (L903–L905), a Strong label depends on whether the historical PR gives key edit locations and reusable implementations. Memory built from trajectories thus does convey the information behind the labels.

Response to W1 (2/3): Trajectory coverage and the GoldPR control

2.2 Direct evidence: trajectory coverage of the target code

For each base task, we count how many of the files and functions changed by its solution patch were modified or read by historical trajectories. The analysis uses Noisy history, since it contains Strong, Weak, and Unrelated tasks. It covers the six sets of historical trajectories that FullTraj and the five memory systems generate on both subsets. Table R2 reports the results.

Table R2: Coverage of the base task's target code by historical trajectories

RelevanceModified filesModified functionsRead filesRead functions
Strong65.7%24.0%76.9%45.9%
Weak69.6%15.3%78.7%44.0%
Unrelated31.7%4.0%38.2%16.8%

The results show three patterns:

  • Trajectories of related history generally reach the target code. Strong and Weak trajectories read about 80% of the target files and modify about two thirds of them. Unrelated trajectories read only 38.2%. The relevance reflected by the labels is thus present in the trajectories.
  • Function-level coverage matches the labels. The share of modified target functions drops from 24.0% for Strong to 15.3% for Weak and 4.0% for Unrelated. Strong and Weak trajectories read similar shares of target functions, but Strong ones modify more. This matches the definitions in Appendix C: Strong history gives key edit locations, while Weak history gives context but not enough to determine the solution.
  • Unrelated history overlaps only on the surface. Unrelated trajectories modify 31.7% of the target files but only 4.0% of the target functions. This agrees with the rule in Appendix C that surface overlap does not count as relevance.

3. Control experiment: memory material identical to the labeled content

To further verify that our conclusions do not depend on this gap, we add a GoldPR experiment. It replaces the trajectories with the content behind the labels:

  • Content: the PR title, description, commit messages, and solution patch of each related historical task.
  • Format: placed in the context in chronological order, replacing the solving trajectories in FullTraj. All other settings stay the same.
  • Scope: the 133 Improvement Tasks under Relevant history, with three runs.

Table R3 compares GoldPR with NoMem, FullTraj, and the five memory systems.

Table R3: Results on Improvement Tasks under Relevant history

SystemResolved (%)Rescue (%)Harm (%)
NoMem35.3 ± 2.7––
FullTraj39.1 ± 0.812.2 [8.9, 15.9]8.4 [5.3, 11.9]
A-Mem40.4 ± 2.411.4 [8.1, 14.8]6.3 [4.0, 9.0]
Mem040.1 ± 2.212.5 [8.8, 16.5]7.8 [5.2, 10.6]
MemOS36.8 ± 2.310.7 [7.7, 13.9]9.2 [6.3, 12.4]
Supermemory40.6 ± 1.311.7 [8.5, 15.2]6.4 [4.3, 8.8]
Claude-Mem41.6 ± 2.312.2 [8.9, 15.7]5.9 [3.8, 8.2]
GoldPR43.9 ± 0.915.0 [11.0, 19.3]6.5 [4.3, 9.0]

Our experiments show that existing trajectories already carry most of the reusable information, and that the main conclusions of the paper still hold:

  • Trajectories already carry most of the reusable information. As you rightly pointed out, trajectories miss some information: GoldPR raises Resolved from 39.1% for FullTraj to 43.9%. Nevertheless, current memory systems already obtain most of this benefit. Claude-Mem achieves 6.3 of the 8.6 points that GoldPR gains over NoMem. The Rescue of each system also reaches 71%–83% of that of GoldPR, with overlapping confidence intervals.
  • The main conclusions of the paper still hold. Even when the historical content is fully correct, memory can still mislead the agent. GoldPR has a Harm of 6.5%, comparable to that of the memory systems.

Response to W1 (3/3): Failure attribution under GoldPR, and summary

4. Further analysis: failure attribution in GoldPR

The analyses above already show that the gap between label objects and memory material does not affect our conclusions. Beyond this, we analyzed the failed GoldPR trajectories to further check the failure attribution in §4.3. GoldPR produces 399 trajectories, of which 224 fail. These failures involve 94 tasks. Using the four failure modes of §4.3, we checked whether each failure was caused by the historical content. Table R4 summarizes the results.

Table R4: Failure modes in the failed GoldPR trajectories

Failure modeFailed trajectoriesShare of failures (%)Tasks
Compatibility mismatch10346.058
Misapplied transfer4017.926
Scope expansion135.812
Failure to revise4017.927
Any mode (counted once)14263.472

Table R4 shows that 63.4% of the failed trajectories exhibit at least one failure mode. Compatibility mismatch is the most common (46.0%). In these cases, the historical fix is correct, but its assumptions do not hold for the current task, and the agent copies it anyway. Section 4.3 already finds this mode more frequent under Relevant history than under the other two. When all history is related and correct, it becomes the dominant mode. This result again supports the conclusion of §4.3: agents tend to treat memory as a ready-made answer rather than reassess it for the current task.

5. Summary

From the design rationale, the trajectory content, and a control experiment, we have shown that the gap between label objects and memory material does not change the conclusions of the paper. The failure analysis of GoldPR further supports the failure attribution in §4.3.

Revisions to the paper

We have added the trajectory coverage analysis, the GoldPR control, and its failure attribution to Appendix J of the revised paper (L2198–L2257, Tables 14–16, in blue).

Reviewer_mxVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new benchmark called "MemDev-Bench", which tries to benchmark whether repository historiesmemory can really help code agents to complete tasks. It evaluate sseveral memory systems under different settings and show some findings, e.g., "Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures".

Critical strengths
  1. The benchmark is well-constructed with real repositories and relevance labels.

  2. It shows some interesting analysis of whether the memory really help or even hurt the quality. It also ablates several settings, e.g., comparison between using memory system and using raw trajectories.

Critical weaknesses
  1. The novelty is quite limited, compared to existing memory benchmarks, and most main findings are not very surprising and well-known. What is the key difference difference from paper "SWE Context Bench: A Benchmark for Context Learning in Coding" and "SWE-Bench-CL: Continual Learning for Coding Agents"?

  2. It is unclear to me what is the benefit of preserving process information, given the super high cost of using full trajectories. We should consider both quality and cost.

  3. The authors have not compared against the latest native memory mechanism in Claude Code or Github Copilot CLI, which have much better design of memory mechanism.

Questions for authors

Refer to Weakness part.

9 comments

Response to W1 (1/2): Differences from SWE-ContextBench and SWE-Bench-CL

Thank you for the careful review and constructive comments! We appreciate your recognition that MemDev-Bench is built on real repositories and provides relevance labels. It is also encouraging that you find value in our analysis of when memory helps or hurts task solving, and in the ablations across settings.

In fact, the paper already discusses both benchmarks. Section 1 (L052–L056) points out their key limitations, and Appendix A (L821–L834) compares how the three benchmarks organize history and handle relevance. Due to space limits, the main text cites the two works only briefly. To make the comparison concrete, this response reports our audit of their released data. The next response then discusses the novelty of our work.

1. Temporal validity: preventing answer leakage

If the history already contains the correct fix for a target task, the agent can reuse the answer instead of transferring experience. Memory gains are then overestimated. Our audit of the released data finds this risk in both benchmarks:

BenchmarkTarget tasks at riskSource of leakage
SWE-ContextBench (full)208/357 (58.3%)117 share the resolving PR with a historical task; for 91, the correct fix already exists in a historical task's starting snapshot
SWE-ContextBench (Lite, used for memory evaluation)64/99 (64.6%)Same two sources as above
SWE-Bench-CL21/273 (7.7%)Tasks are ordered by time, but the correct fix already exists in an earlier task's starting snapshot
MemDev-BenchNoneEach historical PR must be merged or closed before the base PR is created (L148–L150). Backport screening and human review further remove leakage (L165–L168)

Note: The SWE-ContextBench paper reports 376 target tasks. Its released Related file has 362 rows covering 357 distinct target tasks, which our audit counts.

2. Relevance: labeled by actual helpfulness

In both benchmarks, a link between two tasks does not mean that the historical task actually helps the target task. SWE-ContextBench links tasks through issue/PR references with human verification, but it does not grade how much each historical task helps. SWE-Bench-CL defines dependencies only by overlap in modified files. Two examples from the released data:

  • SWE-Bench-CL: django__django-11951 (batch size in bulk insertion) and django__django-13406 (QuerySet deserialization) are marked as dependent only because they modify the same file. Their logic is unrelated, and no fix experience transfers between them.
  • SWE-ContextBench: sympy__sympy-21930 (LaTeX double superscripts) is linked to sympy__sympy-21758 (non-terminating FourierSeries printing). The link comes from a separate issue that the resolving PR happened to fix. The two tasks differ in both root cause and fix goal.

With such links, an evaluation cannot tell whether the history is unhelpful or the memory system fails to exploit relevant history. MemDev-Bench therefore labels each historical task by how much it helps, with three relevance levels and 20 relation types (§2.2; Appendix B). Three additional engineers also labeled 2,270 samples blindly and from scratch. Their labels reach Fleiss' κ of 0.88 for relevance level and 0.80 for relation type (L189–L192).

3. Historical tasks: newly mined, not regrouped

Both benchmarks build histories by regrouping existing SWE-bench tasks. SWE-ContextBench uses SWE-bench tasks as experience tasks (1,100 in its paper; 1,007 in the released data). SWE-Bench-CL arranges 273 SWE-bench Verified tasks into 8 sequences. In contrast, MemDev-Bench mines 4,869 new executable historical tasks from real PR histories and links them to 555 base tasks (L098).

Revisions to the paper

We have added Appendix A.1 (L845–L874, in blue). It reports the leakage audit (Table 6) and the relevance examples above.

Response to W1 (2/2): Novelty of this work

Building on the differences above, this response summarizes our novelty in data construction and in findings.

1. Data construction

Reliable measurement. As shown in the previous response, MemDev-Bench enforces temporal validity and labels history by actual helpfulness. Because answer leakage is prevented, the evaluation can separate "the history is unhelpful" from "the memory system fails to exploit relevant history". The measured effect of memory is thus reliable.

Scalable task construction. Mining executable historical tasks at scale also required a better pipeline. Our open-source pipeline (L548–L550) improves on two weaknesses of existing automated pipelines for SWE-style tasks (§2.3; Appendix D):

  • Reliability: Test and solution patches are split by combining repository test configurations with AST analysis (Appendix D.2.1). Pre- and post-fix environments are built and validated separately (Appendix D.2.2). Both steps keep F2P/P2P results from being distorted.
  • Efficiency: A rule–experience–agent hierarchy invokes an agent only when rules and validated past fixes both fail. This reduces repeated debugging and agent cost (L207–L210; Appendix D.2.3).

These improvements made it possible to build 4,869 executable historical tasks reliably. Beyond our benchmark, the open-source pipeline can also help the community build such tasks more efficiently and accurately.

2. Findings

Prior work has shown that irrelevant history can hurt performance. We take this observation as a starting point. With temporal validity and relevance controlled, our paper answers four questions that existing benchmarks leave open (§1, L078–L086):

QuestionPrior workOur findings
Where do memory gains come from?Both benchmarks report only overall resolve rates.On the 68 tasks that NoMem fails in all three runs, memory rarely expands the set of files the agent locates. Instead, it more consistently improves localization and editing within files: of 18 memory settings, only 2 exceed NoMem in File Recall, while 13 and 16 exceed it in Line Recall and Edit Recall.
How does noise offset memory gains?SWE-ContextBench finds that unfiltered memory brings limited or even negative gains. It reports only overall resolve rates and does not separate gains from regressions at the task level.From Relevant to Noisy history, mean Rescue stays nearly unchanged while mean Harm rises (e.g., on Preservation Tasks, Rescue 3.4%→3.2% and Harm 2.5%→3.7%; L342–L349). Noise thus offsets gains mainly by breaking tasks that were already solvable, not by reducing help on hard tasks.
Why does memory cause failures?SWE-Bench-CL measures how outputs change after irrelevant information is injected, but not functional correctness. Its authors also note that end-to-end evaluation of agent memory has not yet yielded reliable conclusions.We evaluate multiple memory systems end to end. Manual analysis of the 76 tasks that NoMem solves in all three runs confirms 115 memory-induced failures in four failure modes (L403–L413). Compatibility mismatch is more frequent under Relevant history than under the other two (L420–L424), so even related history can be misused.
How does process information change the solving process?SWE-ContextBench compares full trajectories with summaries, but reports only resolve rate, time, and cost. SWE-Bench-CL stores only summarized entries and does not compare them with raw trajectories.Raw trajectories substantially reduce output tokens and steps. We further locate the solving stages and tool types where steps are saved (§4.4, L429–L452; see our response to W2).

Response to W2 (1/2): Benefits of preserving process information

Thank you for this comment. We fully agree that effectiveness and cost should be considered together. For this reason, Tables 2 and 3 report input tokens, output tokens, and steps for every system. Section 4.1 also notes the higher input cost of FullTraj (L355–L357): by Tables 2 and 3, its input tokens are 2.3–7.5× those of NoMem.

We would like to clarify the role of FullTraj. As we now state explicitly in §3 (L286–L287), FullTraj is a long-context reference baseline without compression or selection, not a practical memory design. It helps us understand what memory systems keep and lose when they compress history. The rest of our response examines two questions: (1) what benefits does process information bring, and (2) can these benefits be obtained without FullTraj's input cost? This response covers (1), and the next covers (2).

Process information greatly reduces output and interaction while maintaining quality

DimensionFullTraj
Output tokensAbout 28% fewer than NoMem, and 28%–34% fewer than the five memory systems (L446–L448; Figure 5a)
StepsAbout 34% fewer than NoMem, and 29%–35% fewer than the five memory systems
Source of the reductionOf the 46.2–60.6 steps saved, 24.8–31.3 fall in the edit span. Search and read calls drop by 5.9–8.1 and 5.3–8.9, respectively (L449–L452; Figures 5b and 5c)
Solving qualityOn Improvement Tasks, its resolve rate exceeds NoMem under all three histories (37.3%–39.6% vs. 35.3%; Table 2). On Preservation Tasks under Relevant history, it reaches 97.3%, the highest of all systems (Table 3)

Output tokens are typically priced several times higher than input tokens. Fewer steps also mean fewer model calls and tool executions. These reductions therefore lower both generation cost and latency. All trajectories finish within the interaction budget, so the reductions are not caused by budget truncation.

However, FullTraj does not filter history and is thus more sensitive to noise. On Preservation Tasks, its Harm rises from 0.7% under Relevant history to 4.2% under Noisy history (L349). Overall, process information in history greatly reduces repeated exploration, but placing raw trajectories directly into the context brings both high input cost and noise risk.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: What MemDev-Bench Brings and What We Added

We sincerely thank all reviewers and the Area Chair for their time and thoughtful feedback. We are encouraged that the reviewers find our question “very important and timely” (CjaN), the benchmark “well-constructed with real repositories and relevance labels” (mxVo), and our analyses going “beyond success-rate rankings” (kDWJ). We are also grateful for the reviewers’ insightful suggestions, which have substantially strengthened the paper.

Why this work matters. Memory systems for code agents are multiplying quickly, yet the community still cannot reliably tell whether they truly help. To our knowledge, MemDev-Bench is the first benchmark that ensures both temporally valid histories without answer leakage and graded, human-verified relevance. It is also the first benchmark to provide a fine-grained attribution of memory effects, revealing where gains come from, why failures occur, and what compressed memory loses.

The effort behind it. To this end, we mined and validated 4,869 executable historical tasks from real PR histories, and open-sourced a pipeline that builds such tasks reliably and efficiently. Each relevance label was verified by humans, with independent blind labeling reaching Fleiss’ κ = 0.88. On this foundation, we evaluated 34 memory systems under an information-isolation protocol.

Going beyond what was asked. During the rebuttal, we tried to resolve each concern at its root rather than only answer it. In addition to every requested experiment, we added analyses that we believed would address the underlying question:

AdditionReviewerTypeLocation
Native memory of Claude CodemxVoRequestedAppendix I
Native memory of Codex, with its own no-memory baselinemxVoAdditionalAppendix I
Leakage and relevance audit of SWE-ContextBench and SWE-Bench-CLmxVoAdditionalAppendix A.1
Length-matched Padded historykDWJRequestedAppendix K
GoldPR control, trajectory coverage, and GoldPR failure attributionkDWJAdditionalAppendix J
Validation on unfiltered consecutive PR historiesCjaNRequestedAppendix L
Relevance annotation of 1,200 unfiltered historical tasksCjaNAdditionalAppendix L
Cross-file coverage analysisCjaNRequestedAppendix M
New dependency-based discovery route, piloted on 100 base tasksCjaNAdditionalAppendix M
NoMem + Git history baseline, with memory conditionsCjaNRequestedAppendix N

The findings hold up. Across native memory, correct historical content, length-matched noise, natural PR streams, and open Git history, the main findings of the paper consistently hold: memory mainly sharpens fine-grained localization and editing; regressions offset gains and grow as non-reusable history accumulates; and compressed memory leaves much of the process information unused. These results point to concrete directions for the next generation of memory systems.

We have put our full effort into this work, both in building the benchmark and during the rebuttal. We sincerely hope it can receive a fair and careful evaluation. All revisions are marked in blue in the updated PDF. We would be very happy to answer any further questions before the discussion period closes. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.