acceptodds
Under review as a conference paper at ICLR 2027

CMB: A Coding Memory Benchmark for Long-Horizon and Interleaved Coding Agents

Abstract

Coding agents increasingly operate in long sessions where tasks are interrupted, requirements evolve, branches diverge, and unfinished work must later be resumed. Memory must therefore support subsequent repository actions, not merely recall prior text. We introduce CodeMemoryBench (CMB), an execution-grounded benchmark for memory in long-session coding agents. CMB constructs controlled, interleaved task histories in real repositories, freezes interaction streams and repository checkpoints, and replays identical histories to every memory mechanism. It comprises 216 code-editing tasks and 750 diagnostic questions over 5,595 turns from nine repositories in four languages. Experiments with five existing memory mechanisms reveal substantial room for improvement: the strongest mechanism resolves only 36.6% of CMB tasks with the stronger of two agent backbones. To address this limitation, we propose a three-layer memory that links task facts, reusable experience, and concrete code diffs through provenance and evolution relations. It reaches 43.1% and 35.6% on CMB across the two backbones, compared with 36.6% and 28.2% for the strongest baselines. On SWE-ContextBench, it achieves 47.5% and 28.3%, versus 45.5% and 25.3% for the strongest competing memories. CMB connects diagnostic recall with executable outcomes, measuring whether memory enables agents to resume and complete coding work.

Primary area:
datasets and benchmarks
Keywords:
Agent memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8NBcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CodeMemoryBench(CMB), an execution grounded benchmark for memory in long-session coding agents, which test the agent's long-horizon memory capability under interleaved multitasking in a real repository, including code-editing tasks and diagnostic questions. This paper also proposes TriLink, a three-layer coding memory linking facts, experience, and code evolution. By experiments in this paper, TriLink performs better than other memory methods on CMB and SWE-ContextBench with Claude-Sonnet-5 and Deepseek-v4-flash.

Critical strengths
  • CMB provides a valuable evaluation setting for practically meaningful problems in long-term coding agents that are still insufficiently covered by existing benchmarks. Compared with existing benchmarks, it mainly focuses on whether, after tasks are interrupted and interleaved, an agent can recover historical constraints, identify the target implementation version, and use the relevant memory for subsequent code modifications when returning to a previous task state. CMB combines these requirements into fixed history replay and real repository execution tests, and further distinguishes, through two types of question settings—question answering and code execution—the capabilities in two directions: retrieving and recovering historical information, and whether the model can be supported in returning to a historical state to advance code improvements. Therefore, although some existing benchmarks have already provided evaluation schemes for task interleaving and realistic dialogue scenarios with noise, CMB still has its value for comparing long-term coding memory methods.
  • CMB provides a reusable benchmark construction workflow and, on that basis, forms a relatively complete evaluation resource. Starting from issues, reference patches, and tests in real repositories, this workflow organizes engineering tasks into coding sessions containing multiple rounds of implementation plans, task interleaving, and history-dependent requests, while also saving agent execution traces, code changes, branch information, and repository checkpoints. On this basis, CMB combines historical decision question answering with subsequent code modifications, so that the evaluation can both check whether the agent remembers the relevant information and verify whether that information can support actual repository operations. The authors also used this workflow to build a benchmark covering 9 repositories, 4 programming languages, 216 code tasks, 1,482 technical decision points, and 750 question-answering questions, reflecting considerable data construction and engineering investment.
  • TriLink provides a memory organization method with clear engineering motivation for long-term coding tasks. It stores task state, solution experience, and historical code changes separately, and organizes updates and retrieval through provenance relations and repository anchors, which helps preserve implementation evidence that plain-text summaries tend to lose. Relatively thorough external evaluations and internal ablations provide preliminary support for the practicality of the complete system.
Critical weaknesses
  • The calibration between the memory difficulty and the downstream coding difficulty of the constructed CMB tests is still insufficient. Although the paper provides a FullContext reference, the complete history still contains a large amount of distraction and competing versions, and cannot show what performance the agent could achieve after obtaining correct and sufficient historical evidence. The lack of Only Relevant-History and NoMemory controls makes it difficult for the current results to distinguish the effects of the degree of history dependence, the difficulty of evidence recovery, and coding execution ability, and therefore the evidence for whether the CMB tests effectively target the memory problems in long-term coding agents remains unclear.
  • The distinction from related memory methods remains insufficient. TriLink organizes task facts, reusable experience, and code diffs into a unified system, which has clear engineering feasibility. However, all these components and their combination have already been studied in related work, such as the CodeMem work cited by the authors. Although the paper discusses these works, it lacks targeted comparisons or controlled experiments, so it remains unclear what benefits and improvements TriLink's proposed linking mechanism brings relative to existing designs.
  • The ablation experiments have not yet sufficiently established the contributions of each component and its linking mechanisms. On SWE-ContextBench, using Fact alone outperforms the combination of Fact and Experience, and the experiments do not provide a Fact+Diff configuration, making it impossible to determine the incremental value of Experience in the complete system. In addition, the existing module removals simultaneously change the information supply, retrieval path, and context, and cannot isolate the role of cross-layer linking and updating mechanisms, so the ablation evidence for the overall architecture design is insufficient.
  • The CMB benchmark's representativeness for natural long-term development scenarios remains limited. By synthesizing and replaying interleaved, multi-version histories, CMB focuses on recovering a specified historical implementation; this setting has diagnostic value, but its task distribution and frozen replay protocol are still insufficient to represent more general online engineering development. As the authors discuss, a focus on more realistic conversational scenarios is necessary.
Questions for authors
  • Under the Sonnet configuration, FullContext achieves a task resolution rate of 58.8%, noticeably higher than TriLink's 43.1%; under the DeepSeek configuration, however, the relative performance of the two changes. How do the authors explain this difference? To what extent does it reflect the base model's ability to utilize long histories and distinguish between competing implementation versions?
  • To further distinguish the difficulty of historical evidence recovery from that of agent execution, could the following two controls be added under the same agent framework, initial repository state, and execution budget? 1. Provide only the relevant history: provide the relevant instructions and code evidence accurately selected from the original history that are sufficient to complete the current request, excluding irrelevant tasks and distracting content. 2. Provide no historical memory: same as NoMemory on SWE-ContextBench. Can these results show whether the current failures mainly come from the recovery and understanding of historical evidence?
  • Could comparisons with methods that are closer in design be added to clarify TriLink's new contributions? For example, A-Mem's memory linking and dynamic updating, MAGE's execution-state and dependency maintenance, and CodeMem's code and session memory for iterative development. These methods are respectively related to some of TriLink's core designs, but they have not yet been included in the experimental comparison. Could representative methods suitable for the current evaluation protocol be selected from among them, compared under the same base model, agent framework, and historical input, and their performance along with the corresponding context and memory maintenance costs be reported?
  • Could the roles of the three types of memory and their linking mechanisms be clarified through targeted supplementary ablations? It is suggested to complete the ablation configurations on SWE-ContextBench so as to further clarify the incremental value of each component in other memory testing environments. In addition, it is suggested to keep the same three types of memory content and, under matched retrieval quotas and context budgets, compare link-guided retrieval with independent retrieval at each layer, so as to distinguish the contributions of information complementarity and the linking mechanism.
4 comments

W1: Without Only Relevant-History and NoMemory controls, it is unclear whether failures come from history dependence, evidence recovery, or coding ability.

R1: NoMemory is uninformative on CMB by construction: final requests refer to an earlier revision only by position, contrast or description (e.g., "go back to the first version"), and neither the request nor the workspace identifies that implementation. For Only Relevant-History, we inject only the current task's raw development turns. CMB Resolved (%):

MethodSonnet-5v4-flash
TriLink43.135.6
FullContext58.828.2
Only Relevant-History63.538.7

The gap to TriLink shows remaining headroom for memory.

W2: The distinction from related methods (A-Mem, MAGE, CodeMem) is insufficient, and direct comparisons are missing.

R2: CodeMem tracks per-function AST diffs within a single-task session to prevent forgetting, treating the latest edit as correct; CMB requires recovering a specific earlier alternative (86.6% of points) across interleaved tasks and branches. A-Mem links homogeneous notes by similarity and rewrites older notes; TriLink links typed records by provenance and keeps superseded facts recoverable. MAGE manages the online state of one task and stores no code changes, so it does not fit frozen multi-task replay. We add CodeMem (session memory as context provider) and A-Mem to CMB with v4-flash:

MethodResolved (%)
A-Mem23.1
CodeMem28.7
TriLink35.6

CodeMem is the strongest baseline, consistent with its code-centric diff history, but TriLink still improves on it by 6.9 points.

W3: The ablations do not isolate Experience (no Fact + Diff on SWE-ContextBench) or the linking mechanism, since layer removal also changes retrieval and context.

R3: We add Fact + Diff on SWE-ContextBench (v4-flash):

ConfigurationResolved (%)
Fact only23.2
Fact + Experience21.2
Fact + Diff25.3
Full TriLink28.3

Experience adds 3.0 points on top of Fact + Diff, but not without diff evidence. To isolate linking, we keep all three layers with matched quotas and context budgets, and replace link-guided retrieval with independent retrieval at each layer (v4-flash):

RetrievalCMBSWE-ContextBench
Independent32.426.3
Link-guided (TriLink)35.628.3

With the same underlying memory store, matched retrieval quotas, and context budgets, link-guided retrieval adds 3.2 / 2.0 points, isolating the contribution of linking from information complementarity.

W4: Synthetic, frozen histories may not represent natural online development.

R4: We agree that synthetic histories cannot fully capture natural developer–agent interactions. CMB is designed as a controlled benchmark for one practically relevant challenge: recovering prior decisions across interleaved tasks and interrupted development. The repositories, source tasks, and tests are real, while the interaction sequences and coding trajectories are model-generated. Beyond CMB, TriLink also outperforms competing memory baselines on SWE-ContextBench, which evaluates a non-interleaved setting. Evaluating on naturally occurring developer–agent logs is an important direction for future work.

W5: Why does FullContext beat TriLink with Sonnet-5 but not with flash?

R5: FullContext keeps up to 500K tokens: current-task turns first, then unrelated turns and noise. Removing the unrelated part (R1) gains 4.7 points for Sonnet-5 but 10.5 for flash, so the weaker backbone is far more sensitive to long-context distraction. TriLink injects only selected evidence and depends less on this ability.

Thank you again for your thoughtful review. If our responses have addressed your concerns, we would be grateful if you could consider raising your score.

Reviewer_8NBcOpenReview ↗

Follow-up on numerical consistency and evaluation denominators

Thank you for the additional experiments. While these additions address several of my original requests, I found numerical inconsistencies that need to be resolved before I can reassess the evidence.

First, Table 8 reports Sonnet-5/RepoMemory success rates of 8%, 25%, and 16% over groups of 12, 88, and 112 tasks. These imply 1, 22, and 18 successful tasks, respectively: 41 successes within the 212-task subset. However, Table 2 reports only 38 successes across all 216 tasks. Rounding and the four excluded tasks cannot reconcile these figures.

Second, the Sonnet-5/OpenViking row in Table 9 reports Patch N/A, task-level F2P, and task-level P2P rates of 15.8%, 33.5%, and 67.0%. None corresponds to an integer count out of 216 after rounding. Please clarify the denominators and any exclusion rules used for these metrics.

Third, the new Only Relevant-History results of 63.5% and 38.7% likewise do not correspond to single-run integer success counts out of 216. Please provide the counts, denominators, number of runs, and aggregation procedure.

Please reconcile these results using the underlying task-level outcomes and clarify whether the tables use the same evaluation runs and task sets. These issues affect the consistency and interpretability of the reported evidence. I therefore maintain my current rating at this stage.

We thank the reviewer for checking the numbers so carefully. All three inconsistencies are real, and we have traced each of them; the causes are explained under each point below. All CMB results use pass@2 for every method: each task is attempted twice independently and counts as resolved if at least one attempt passes all F2P and P2P tests. Rates are integer counts over 216; there is no averaging across runs. All corrected rates below are computed over the full task set.

W1: Table 8 reports Sonnet-5/RepoMemory success rates of 8%, 25%, and 16% over groups of 12, 88, and 112 tasks. These imply 1, 22, and 18 successful tasks, respectively: 41 successes within the 212-task subset. However, Table 2 reports only 38 successes across all 216 tasks.

R1: RepoMemory has no official open-source implementation, so we re-implemented it from the paper and ran several tuning rounds to reproduce it faithfully. Table 8 reports the final configuration, but the Sonnet-5/RepoMemory row in Table 2 (and Table 9) was not updated and still reflects an earlier tuning round (38/216). Under the final configuration, RepoMemory resolves 1/12, 22/88 and 18/112 tasks in the three revision groups and none of the 4 remaining tasks, i.e., 41/216 = 19.0% Resolved (previously 38/216 = 17.6%), and its QA score is 20.8% (previously 18.7%). We will update Tables 2 and 9 accordingly and describe the re-implementation in the appendix. RepoMemory remains the lowest-scoring method with Sonnet-5, so no comparison in the paper changes.

W2: The Sonnet-5/OpenViking row in Table 9 reports Patch N/A, task-level F2P, and task-level P2P rates of 15.8%, 33.5%, and 67.0%. None corresponds to an integer count out of 216 after rounding. Please clarify the denominators and any exclusion rules used for these metrics.

R2: When grading this run, the Docker test container of one task ran out of memory and returned no test report. When we compiled the statistics, this task was not counted in the denominator of the three detailed metrics, so they were computed over 215 tasks, whereas Resolved was computed over all 216 tasks with this task counted as unresolved, as Eq. 2 specifies. The reported values therefore correspond to 34/215, 72/215 and 144/215. We re-graded the task with sufficient container memory: it produces a patch that fails its tests, so it remains unresolved, and all task-level metrics now use 216 tasks.

MetricReported (%)Corrected countCorrected (%)
Patch N/A15.834 / 21615.7
Tasks-F2P33.572 / 21633.3
Tasks-P2P67.0144 / 21666.7
Resolved29.263 / 21629.2

Each correction is within 0.5 points and does not affect any comparison in the paper.

W3: The new Only Relevant-History results of 63.5% and 38.7% likewise do not correspond to single-run integer success counts out of 216. Please provide the counts, denominators, number of runs, and aggregation procedure.

R3: Like all CMB results, these use pass@2: two independent attempts per task, and a task counts as resolved if either attempt passes all tests. Resolved is the number of resolved tasks divided by 216, and an attempt without a valid test report counts as failing. In these runs, the Docker test containers of 5 (Sonnet-5) and 4 (flash) tasks ran out of memory during grading and returned no test report. When we compiled the statistics, these tasks were not counted in the denominator, so the reported figures were 134/211 and 82/212. After re-grading these tasks with sufficient container memory, none of them is resolved, so the counts are unchanged and only the denominator is corrected:

BackboneReported (%)Corrected countCorrected (%)
Sonnet-563.5 (134/211)134 / 21662.0
v4-flash38.7 (82/212)82 / 21638.0

The corrected values remain above TriLink (43.1% / 35.6%) and FullContext (58.8% / 28.2%) for both backbones, so the conclusions in our earlier responses are unchanged.

We are sincerely grateful to the reviewer for checking our numbers with such care. Tracing these inconsistencies led us to correct how failed grading was counted, which has made the work more rigorous. This careful review has been one of the most valuable outcomes of our ICLR submission.

Reviewer_JpJCOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces CMB, a benchmark for memory in long, interleaved coding sessions built from SWE-bench Pro tasks, with test-graded code tasks and QA questions. It also proposes TriLink, a memory with linked fact, experience and code-diff layers, which outperforms existing memory systems on CMB and improves on SWE-ContextBench.

Critical strengths
  • Studies an important and timely problem: coding agents in long sessions with interruptions, branch switches and changing requirements.
  • Strong evaluation design. Frozen replay gives a fair comparison, and test-based grading ties memory to real coding outcomes.
  • TriLink is well motivated, since it keeps code diffs that summaries lose. It helps on both CMB and an external benchmark, across two backbones.
  • Good analysis: layer ablations, results by revision depth, and injected-token efficiency.
Critical weaknesses
  • With the stronger backbone, full raw history outperforms all memory methods, as Table 2 shows. A cost comparison and a setting where the full history does not fit would make the case for memory clearer.
  • Sessions are synthetic and revision-heavy, with few noise turns. It is unclear how well they match real developer sessions.
  • MemoryCode [1], which studies instruction tracking across multi-session coding with distractors and updates, and SWE-Exp [2], which reuses experience from past trajectories, should be discussed.
Questions for authors
  • How does TriLink perform without CMB-provided metadata such as branch and code object?
  • How well does the QA judge agree with human labels?
1 comment

We sincerely thank the reviewer for the positive assessment and constructive suggestions.

W1: Sessions are synthetic and revision-heavy, with few noise turns. It is unclear how well they match real developer sessions.

R1: We agree that synthetic histories cannot fully capture natural developer–agent interactions. CMB is designed as a controlled benchmark for one practically relevant challenge: recovering prior decisions across interleaved tasks and interrupted development. The repositories, source tasks, and tests are real, while the interaction sequences and coding trajectories are model-generated. Beyond CMB, TriLink also outperforms competing memory baselines on SWE-ContextBench, which evaluates a non-interleaved setting. Evaluating on naturally occurring developer–agent logs is an important direction for future work.

W2: MemoryCode, which studies instruction tracking across multi-session coding with distractors and updates, and SWE-Exp, which reuses experience from past trajectories, should be discussed.

R2: We will cite and discuss both papers in Section 2. MemoryCode tracks simple coding conventions across mentor–mentee sessions, where only the latest update is relevant, and checks compliance with regular expressions. CMB instead requires recovering a specific earlier revision (86.6% of points) and grades it with repository tests. SWE-Exp distills comprehension and modification experience from past trajectories for new issues, which relates to our Experience layer; TriLink additionally links it to task state and ordered code diffs within a session.

W3: How does TriLink perform without CMB-provided metadata such as branch and code object?

R3: CMB point and segment identifiers are used only at ingestion to group diffs and look up checkpoints, never at query time, and retrieval receives no target-branch filter. Branch names come from the recorded trace itself. Without CMB metadata, grouping diffs only by branch, file, symbol and line region, TriLink reaches 32.4% Resolved with v4-flash, versus 35.6% with metadata, and remains above all memory baselines.

W4: How well does the QA judge agree with human labels?

R4: We randomly sampled 30% of the QA questions (225 of 750), and two authors blind to method identity and judge decisions labeled whether each requirement is covered. The judge agrees with human labels at Cohen's κ = 0.83 on requirement-level decisions. We also re-scored all answers with a second judge (gpt-5.6-sol): scores are 1.0 points lower on average and the method ranking is unchanged (see our response to Reviewer EkNJ).

Reviewer_EkNJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CMB is an execution-grounded benchmark for memory in long, interleaved coding sessions. SWE-bench Pro tasks with disjoint edit scopes are decomposed into 1,482 decision points with 3–5 alternative revisions each. Task streams are interleaved with noise turns, executed, and frozen with repository checkpoints (5,595 turns, 9 repositories, 4 languages). Memory mechanisms ingest the identical stream and are graded on 216 test-based code tasks and 750 QA questions. The paper also proposes TriLink, which links fact, experience, and diff memory and resolves 43.1% / 35.6% of tasks against 36.6% / 28.2% for the strongest baseline.

Critical strengths
  1. Grading memory by whether the resulting edit passes F2P/P2P tests is a valuable and underexplored target.
  2. Frozen histories replayed identically to every mechanism give a controlled comparison.
  3. The dual QA/execution design is justified by the two tracks ranking methods differently.
  4. Construction is careful and auditable with file-disjoint tasks, coverage checks, and reference-form and revision statistics.
Critical weaknesses
  1. There is no NoMemory or oracle-recall condition on CMB, so it is unknown whether tasks need memory or are solvable with perfect recall.
  2. The agent starts from a clone of the frozen repository, so earlier revisions may be recoverable from git without any memory.
  3. Raw full history with Sonnet-5 resolves 58.8%, above every memory method, yet this is excluded from the headline framing.
  4. TriLink receives recorded patches, checkpoint diffs, and CMB metadata that the text-only baselines do not.
Questions for authors

Please address the following questions:

  1. Can authors add NoMemory and oracle results on CMB?
  2. Does the cloned repository expose construction branches or commits, and do agents use git history?
  3. How does FullContext fit 0.5–1.2M-token histories, and why is FullContext/Sonnet-5 missing?
  4. In the "Fact Memory" ablation, how are diff chains located, given that facts anchor diff retrieval?
  5. Which model is the QA judge, and how stable are QA scores across judge runs?
4 comments

W1: Can authors add NoMemory and oracle results on CMB?

R1: NoMemory is uninformative on CMB by construction. Each final request refers to an earlier revision only by position, contrast or description, for example "go back to the first version". Neither the instruction nor the workspace says which implementation that is. Without memory, the agent cannot recover the intended revision.

For the oracle, we inject only the raw development turns of the current task, excluding all other tasks and noise turns. CMB Resolved (%):

MethodSonnet-5v4-flash
TriLink43.135.6
FullContext58.828.2
Oracle (current task only)62.038.0

The gap to TriLink shows remaining headroom for memory.

W2: Does the cloned repository expose construction branches or commits, and do agents use git history?

R2: No. The evaluation workspace exposes only the final state of each feature branch, not how that state was built.

  • No construction history in git. Each feat/<task> branch has exactly two commits: a shallow clone of base_commit with no ancestors, and one wip commit holding the frozen end-of-development tree. Upstream branches, tags and remotes are removed, and unreachable objects are garbage-collected. Neither per-round versions nor later upstream changes can be recovered.
  • No identifying cues. Commit messages are fixed strings with no task, point or version IDs. HEAD starts on a neutral branch, and each finalize request describes the feature semantically without naming its branch.
  • No trail across points. Finalization edits are folded into the same wip commit, so a later point cannot see an earlier point's work through git.

W3: How does FullContext fit 0.5–1.2M-token histories, and why is FullContext/Sonnet-5 missing?

R3: FullContext keeps at most 500K tokens of history. Turns from the current task are placed first, and the remaining budget is filled with unrelated turns. We will state this in Section 5.1. FullContext/Sonnet-5 is reported in Table 2 (58.8 Code / 56.1 QA), Table 3 and Table 9; we will also add it to Table 8 and Figure 4.

W4: In the "Fact Memory" ablation, how are diff chains located, given that facts anchor diff retrieval?

R4: In "w/o Fact Memory", facts are still retrieved and still locate the diff chains by branch, falling back to file. Only the standalone fact entries are removed from the injected context. The variant therefore measures the contribution of fact text, not of fact anchoring. We will rename it "w/o fact entries" and clarify this in Section 5.3.

W5: Which model is the QA judge, and how stable are QA scores across judge runs?

R5: The QA judge in all reported results is claude-sonnet-4-6 with temperature 0 and thinking disabled. We will state this in Section 3.1 and Appendix A.3.1. To check that the conclusions do not depend on this judge, we re-scored the same frozen agent answers with a second judge, gpt-5.6-sol.

QA scores under a second judge (%).

BackboneMethodclaude-sonnet-4-6gpt-5.6-sol
Sonnet-5FullContext56.155.2
RAG48.046.3
Mem037.835.4
MemOS51.949.3
OpenViking46.845.1
RepoMemory18.719.2
TriLink58.258.4
v4-flashFullContext53.451.6
RAG62.261.2
Mem040.041.7
MemOS51.449.5
OpenViking38.637.2
RepoMemory9.67.6
TriLink64.264.6

Scores under gpt-5.6-sol are 1.1 points lower on average, and the ranking order of the methods does not change.

Thank you again for your thoughtful review. If our responses have addressed your concerns, we would be grateful if you could consider raising your score.

Thanks to the authors for clarifying the issues. I would like to raise the score from 2 to 3.

Thank you very much for your response! We truly appreciate your valuable feedback and will incorporate your suggestions into future versions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.