- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CodeMemoryBench(CMB), an execution grounded benchmark for memory in long-session coding agents, which test the agent's long-horizon memory capability under interleaved multitasking in a real repository, including code-editing tasks and diagnostic questions. This paper also proposes TriLink, a three-layer coding memory linking facts, experience, and code evolution. By experiments in this paper, TriLink performs better than other memory methods on CMB and SWE-ContextBench with Claude-Sonnet-5 and Deepseek-v4-flash.
Critical strengths
- CMB provides a valuable evaluation setting for practically meaningful problems in long-term coding agents that are still insufficiently covered by existing benchmarks. Compared with existing benchmarks, it mainly focuses on whether, after tasks are interrupted and interleaved, an agent can recover historical constraints, identify the target implementation version, and use the relevant memory for subsequent code modifications when returning to a previous task state. CMB combines these requirements into fixed history replay and real repository execution tests, and further distinguishes, through two types of question settings—question answering and code execution—the capabilities in two directions: retrieving and recovering historical information, and whether the model can be supported in returning to a historical state to advance code improvements. Therefore, although some existing benchmarks have already provided evaluation schemes for task interleaving and realistic dialogue scenarios with noise, CMB still has its value for comparing long-term coding memory methods.
- CMB provides a reusable benchmark construction workflow and, on that basis, forms a relatively complete evaluation resource. Starting from issues, reference patches, and tests in real repositories, this workflow organizes engineering tasks into coding sessions containing multiple rounds of implementation plans, task interleaving, and history-dependent requests, while also saving agent execution traces, code changes, branch information, and repository checkpoints. On this basis, CMB combines historical decision question answering with subsequent code modifications, so that the evaluation can both check whether the agent remembers the relevant information and verify whether that information can support actual repository operations. The authors also used this workflow to build a benchmark covering 9 repositories, 4 programming languages, 216 code tasks, 1,482 technical decision points, and 750 question-answering questions, reflecting considerable data construction and engineering investment.
- TriLink provides a memory organization method with clear engineering motivation for long-term coding tasks. It stores task state, solution experience, and historical code changes separately, and organizes updates and retrieval through provenance relations and repository anchors, which helps preserve implementation evidence that plain-text summaries tend to lose. Relatively thorough external evaluations and internal ablations provide preliminary support for the practicality of the complete system.
Critical weaknesses
- The calibration between the memory difficulty and the downstream coding difficulty of the constructed CMB tests is still insufficient. Although the paper provides a FullContext reference, the complete history still contains a large amount of distraction and competing versions, and cannot show what performance the agent could achieve after obtaining correct and sufficient historical evidence. The lack of Only Relevant-History and NoMemory controls makes it difficult for the current results to distinguish the effects of the degree of history dependence, the difficulty of evidence recovery, and coding execution ability, and therefore the evidence for whether the CMB tests effectively target the memory problems in long-term coding agents remains unclear.
- The distinction from related memory methods remains insufficient. TriLink organizes task facts, reusable experience, and code diffs into a unified system, which has clear engineering feasibility. However, all these components and their combination have already been studied in related work, such as the CodeMem work cited by the authors. Although the paper discusses these works, it lacks targeted comparisons or controlled experiments, so it remains unclear what benefits and improvements TriLink's proposed linking mechanism brings relative to existing designs.
- The ablation experiments have not yet sufficiently established the contributions of each component and its linking mechanisms. On SWE-ContextBench, using Fact alone outperforms the combination of Fact and Experience, and the experiments do not provide a Fact+Diff configuration, making it impossible to determine the incremental value of Experience in the complete system. In addition, the existing module removals simultaneously change the information supply, retrieval path, and context, and cannot isolate the role of cross-layer linking and updating mechanisms, so the ablation evidence for the overall architecture design is insufficient.
- The CMB benchmark's representativeness for natural long-term development scenarios remains limited. By synthesizing and replaying interleaved, multi-version histories, CMB focuses on recovering a specified historical implementation; this setting has diagnostic value, but its task distribution and frozen replay protocol are still insufficient to represent more general online engineering development. As the authors discuss, a focus on more realistic conversational scenarios is necessary.
Questions for authors
- Under the Sonnet configuration, FullContext achieves a task resolution rate of 58.8%, noticeably higher than TriLink's 43.1%; under the DeepSeek configuration, however, the relative performance of the two changes. How do the authors explain this difference? To what extent does it reflect the base model's ability to utilize long histories and distinguish between competing implementation versions?
- To further distinguish the difficulty of historical evidence recovery from that of agent execution, could the following two controls be added under the same agent framework, initial repository state, and execution budget? 1. Provide only the relevant history: provide the relevant instructions and code evidence accurately selected from the original history that are sufficient to complete the current request, excluding irrelevant tasks and distracting content. 2. Provide no historical memory: same as NoMemory on SWE-ContextBench. Can these results show whether the current failures mainly come from the recovery and understanding of historical evidence?
- Could comparisons with methods that are closer in design be added to clarify TriLink's new contributions? For example, A-Mem's memory linking and dynamic updating, MAGE's execution-state and dependency maintenance, and CodeMem's code and session memory for iterative development. These methods are respectively related to some of TriLink's core designs, but they have not yet been included in the experimental comparison. Could representative methods suitable for the current evaluation protocol be selected from among them, compared under the same base model, agent framework, and historical input, and their performance along with the corresponding context and memory maintenance costs be reported?
- Could the roles of the three types of memory and their linking mechanisms be clarified through targeted supplementary ablations? It is suggested to complete the ablation configurations on SWE-ContextBench so as to further clarify the incremental value of each component in other memory testing environments. In addition, it is suggested to keep the same three types of memory content and, under matched retrieval quotas and context budgets, compare link-guided retrieval with independent retrieval at each layer, so as to distinguish the contributions of information complementarity and the linking mechanism.
4 comments
Thank you again for your thoughtful review. If our responses have addressed your concerns, we would be grateful if you could consider raising your score.
Follow-up on numerical consistency and evaluation denominators
Thank you for the additional experiments. While these additions address several of my original requests, I found numerical inconsistencies that need to be resolved before I can reassess the evidence.
First, Table 8 reports Sonnet-5/RepoMemory success rates of 8%, 25%, and 16% over groups of 12, 88, and 112 tasks. These imply 1, 22, and 18 successful tasks, respectively: 41 successes within the 212-task subset. However, Table 2 reports only 38 successes across all 216 tasks. Rounding and the four excluded tasks cannot reconcile these figures.
Second, the Sonnet-5/OpenViking row in Table 9 reports Patch N/A, task-level F2P, and task-level P2P rates of 15.8%, 33.5%, and 67.0%. None corresponds to an integer count out of 216 after rounding. Please clarify the denominators and any exclusion rules used for these metrics.
Third, the new Only Relevant-History results of 63.5% and 38.7% likewise do not correspond to single-run integer success counts out of 216. Please provide the counts, denominators, number of runs, and aggregation procedure.
Please reconcile these results using the underlying task-level outcomes and clarify whether the tables use the same evaluation runs and task sets. These issues affect the consistency and interpretability of the reported evidence. I therefore maintain my current rating at this stage.
We thank the reviewer for checking the numbers so carefully. All three inconsistencies are real, and we have traced each of them; the causes are explained under each point below. All CMB results use pass@2 for every method: each task is attempted twice independently and counts as resolved if at least one attempt passes all F2P and P2P tests. Rates are integer counts over 216; there is no averaging across runs. All corrected rates below are computed over the full task set.
W1: Table 8 reports Sonnet-5/RepoMemory success rates of 8%, 25%, and 16% over groups of 12, 88, and 112 tasks. These imply 1, 22, and 18 successful tasks, respectively: 41 successes within the 212-task subset. However, Table 2 reports only 38 successes across all 216 tasks.
R1: RepoMemory has no official open-source implementation, so we re-implemented it from the paper and ran several tuning rounds to reproduce it faithfully. Table 8 reports the final configuration, but the Sonnet-5/RepoMemory row in Table 2 (and Table 9) was not updated and still reflects an earlier tuning round (38/216). Under the final configuration, RepoMemory resolves 1/12, 22/88 and 18/112 tasks in the three revision groups and none of the 4 remaining tasks, i.e., 41/216 = 19.0% Resolved (previously 38/216 = 17.6%), and its QA score is 20.8% (previously 18.7%). We will update Tables 2 and 9 accordingly and describe the re-implementation in the appendix. RepoMemory remains the lowest-scoring method with Sonnet-5, so no comparison in the paper changes.
W2: The Sonnet-5/OpenViking row in Table 9 reports Patch N/A, task-level F2P, and task-level P2P rates of 15.8%, 33.5%, and 67.0%. None corresponds to an integer count out of 216 after rounding. Please clarify the denominators and any exclusion rules used for these metrics.
R2: When grading this run, the Docker test container of one task ran out of memory and returned no test report. When we compiled the statistics, this task was not counted in the denominator of the three detailed metrics, so they were computed over 215 tasks, whereas Resolved was computed over all 216 tasks with this task counted as unresolved, as Eq. 2 specifies. The reported values therefore correspond to 34/215, 72/215 and 144/215. We re-graded the task with sufficient container memory: it produces a patch that fails its tests, so it remains unresolved, and all task-level metrics now use 216 tasks.
| Metric | Reported (%) | Corrected count | Corrected (%) |
|---|---|---|---|
| Patch N/A | 15.8 | 34 / 216 | 15.7 |
| Tasks-F2P | 33.5 | 72 / 216 | 33.3 |
| Tasks-P2P | 67.0 | 144 / 216 | 66.7 |
| Resolved | 29.2 | 63 / 216 | 29.2 |
Each correction is within 0.5 points and does not affect any comparison in the paper.
W3: The new Only Relevant-History results of 63.5% and 38.7% likewise do not correspond to single-run integer success counts out of 216. Please provide the counts, denominators, number of runs, and aggregation procedure.
R3: Like all CMB results, these use pass@2: two independent attempts per task, and a task counts as resolved if either attempt passes all tests. Resolved is the number of resolved tasks divided by 216, and an attempt without a valid test report counts as failing. In these runs, the Docker test containers of 5 (Sonnet-5) and 4 (flash) tasks ran out of memory during grading and returned no test report. When we compiled the statistics, these tasks were not counted in the denominator, so the reported figures were 134/211 and 82/212. After re-grading these tasks with sufficient container memory, none of them is resolved, so the counts are unchanged and only the denominator is corrected:
| Backbone | Reported (%) | Corrected count | Corrected (%) |
|---|---|---|---|
| Sonnet-5 | 63.5 (134/211) | 134 / 216 | 62.0 |
| v4-flash | 38.7 (82/212) | 82 / 216 | 38.0 |
The corrected values remain above TriLink (43.1% / 35.6%) and FullContext (58.8% / 28.2%) for both backbones, so the conclusions in our earlier responses are unchanged.
We are sincerely grateful to the reviewer for checking our numbers with such care. Tracing these inconsistencies led us to correct how failed grading was counted, which has made the work more rigorous. This careful review has been one of the most valuable outcomes of our ICLR submission.
W1: Without Only Relevant-History and NoMemory controls, it is unclear whether failures come from history dependence, evidence recovery, or coding ability.
R1: NoMemory is uninformative on CMB by construction: final requests refer to an earlier revision only by position, contrast or description (e.g., "go back to the first version"), and neither the request nor the workspace identifies that implementation. For Only Relevant-History, we inject only the current task's raw development turns. CMB Resolved (%):
The gap to TriLink shows remaining headroom for memory.
W2: The distinction from related methods (A-Mem, MAGE, CodeMem) is insufficient, and direct comparisons are missing.
R2: CodeMem tracks per-function AST diffs within a single-task session to prevent forgetting, treating the latest edit as correct; CMB requires recovering a specific earlier alternative (86.6% of points) across interleaved tasks and branches. A-Mem links homogeneous notes by similarity and rewrites older notes; TriLink links typed records by provenance and keeps superseded facts recoverable. MAGE manages the online state of one task and stores no code changes, so it does not fit frozen multi-task replay. We add CodeMem (session memory as context provider) and A-Mem to CMB with v4-flash:
CodeMem is the strongest baseline, consistent with its code-centric diff history, but TriLink still improves on it by 6.9 points.
W3: The ablations do not isolate Experience (no Fact + Diff on SWE-ContextBench) or the linking mechanism, since layer removal also changes retrieval and context.
R3: We add Fact + Diff on SWE-ContextBench (v4-flash):
Experience adds 3.0 points on top of Fact + Diff, but not without diff evidence. To isolate linking, we keep all three layers with matched quotas and context budgets, and replace link-guided retrieval with independent retrieval at each layer (v4-flash):
With the same underlying memory store, matched retrieval quotas, and context budgets, link-guided retrieval adds 3.2 / 2.0 points, isolating the contribution of linking from information complementarity.
W4: Synthetic, frozen histories may not represent natural online development.
R4: We agree that synthetic histories cannot fully capture natural developer–agent interactions. CMB is designed as a controlled benchmark for one practically relevant challenge: recovering prior decisions across interleaved tasks and interrupted development. The repositories, source tasks, and tests are real, while the interaction sequences and coding trajectories are model-generated. Beyond CMB, TriLink also outperforms competing memory baselines on SWE-ContextBench, which evaluates a non-interleaved setting. Evaluating on naturally occurring developer–agent logs is an important direction for future work.
W5: Why does FullContext beat TriLink with Sonnet-5 but not with flash?
R5: FullContext keeps up to 500K tokens: current-task turns first, then unrelated turns and noise. Removing the unrelated part (R1) gains 4.7 points for Sonnet-5 but 10.5 for flash, so the weaker backbone is far more sensitive to long-context distraction. TriLink injects only selected evidence and depends less on this ability.