- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
ContextGraph stores lessons from coding-agent trajectories in a graph, lets agents select relevant lessons during subsequent tasks, and updates memory after each attempt. The authors evaluate the system across several benchmarks and model backbones, reporting improvements over agents without memory and existing memory systems.
Critical strengths
The paper investigates the useful, practical question of how coding agents can reuse experience across tasks. The eval includes several memory baselines and models, and the ablations provide useful evidence that abstracting trajectories and selecting lessons dynamically matter.
Critical weaknesses
W1. The presentation makes the method unnecessarily difficult to assess. The paper repeatedly states its takeaways, while important operational details remain unclear. For example, what qualifies a lesson for storage, how unreliable or contradictory lessons are handled, and whether memory consultation is enforced or merely encouraged. Equation 3 introduces (m_j), which is briefly revisited in Appendix F but contributes little beyond listing tool actions. The different memory pools and evaluation configurations are also difficult to track. Figure 3 is overcrowded and hard to read; Figure 4(b) has an overflowing “+13.7” annotation. The latter is minor, but the broader lack of clarity substantially affects my ability to assess the method.
W2. The contribution and evidence do not sufficiently justify the proposed complexity. The method combines lesson extraction, graph storage, agent-directed retrieval, and online updates. The paper does not clearly establish what substantial new insight this combination provides beyond existing memory approaches. Removing graph links retains 31.1% resolution versus 35.1% for the full system, suggesting that much of the benefit survives without the graph. Moreover, the reported confidence intervals for improvements over the strongest baseline on each of the three main benchmarks include zero. This does not establish that the improvements are absent, but it limits the strength of the comparative claims. Staleness and erroneous memory accumulation also remain untested despite their relevance to a continually growing memory.
Questions for authors
Q1. What are the actual criteria for admitting, revising, and removing lessons? How does the system handle advice that becomes stale after repository changes, conflicts with another lesson, or was inferred incorrectly from a failed attempt?
Q2. Are the memory consultations before planning and after errors enforced by the harness? Do the logs quantify whether agents read and subsequently apply the retrieved lessons?
Q3. Can you report variability across the repeated runs, completed-evaluation counts for each method, and a paired analysis of the principal comparisons? How does the analysis account for tasks sharing an evolving memory?
Q4. What evidence distinguishes the benefit of general debugging advice from repository-specific or recently acquired knowledge? The static-playbook ablation is useful, but it does not establish when external memory is preferable to learning general strategies in the model.
Q5. What access boundaries are assumed for cross-repository memory? Removing repository names does not necessarily remove confidential content, particularly when source records and online lessons retain files and commands. Is the intended scope restricted to repositories with shared access permissions?
4 comments
W2. "The paper does not clearly establish what substantial new insight this combination provides beyond existing memory approaches. Removing graph links retains 31.1% resolution versus 35.1% for the full system, suggesting that much of the benefit survives without the graph."
Thank you for this point. Section 1 now states the insight directly. The insight is which parts of experience memory carry the gain. Existing memory systems each combine several design choices, and under one agent, backbone, and budget they range from -2.1 to +9.9 points over no memory (Table 7A). Only an ablation that removes one choice at a time can tell which choice matters (Table R2).
Table R2. Ablation on Related-Lite99: one choice removed at a time from the full system, which reaches 35.1% (five-run means from Table 3; drops in points, tested with the two analyses of Table R4; bold marks results that exclude zero).
| Variant | Resolution | Drop from full | Paired 95% interval | Run-level Welch test |
|---|---|---|---|---|
| Raw trajectories, no abstraction | 20.5% | 14.6 | 10.0 to 19.2 | p < 0.001 |
| Static playbook, no matching to the task | 20.8% | 14.3 | 9.7 to 19.0 | p < 0.001 |
| Fixed top-k by embedding similarity | 24.3% | 10.8 | 6.5 to 15.3 | p = 0.001 |
| Frozen pool, no online updates | 28.7% | 6.4 | 1.5 to 11.4 | p = 0.011 |
| No graph links | 31.1% | 4.0 | -1.0 to 9.1 | p = 0.07 |
| Planning-only consultation | 33.2% | 1.9 | -2.6 to 6.6 | p = 0.37 |
Four choices are supported under both analyses, and each has a clear reason. Raw trajectories lose the entire gain because an excerpt from another repository carries that repository's code, whereas an abstract lesson carries the decision and the condition under which it applies (Section 4.3). The static playbook loses it because every task receives the same three general lessons, so they are not matched to the task (Section 4.4). Fixed top-k loses most of the gain because the three nearest lessons are handed over once, whereas the agent in the full system searches memory itself and keeps the lessons that fit. Freezing the pool removes the lessons learned during the stream. The first three are the findings kindly noted in the review ("abstracting trajectories and selecting lessons dynamically matter"), and they now have paired support. The drops for graph links and for consultation after errors are not supported, and the caption of Table 3 and Appendix H.6 now say so.
On the graph links, we agree with the reviewer's reading that most of the benefit survives without them. Section 3.1 now describes them as used for retrieval and for tracing each lesson back to its source.
W2 (continued). "Staleness and erroneous memory accumulation also remain untested despite their relevance to a continually growing memory."
Thank you for highlighting this important point for a growing memory. Both effects are present in every run we report, and the gain stays at a similar size as lessons accumulate (Table R3). Every task that ends with a patch adds lessons whether or not the patch is correct, no test result enters memory, and nothing is removed, so lessons from failed attempts build up in every run. Tasks arrive in order of issue creation time (Section 3), so the later tasks of a repository are also served by lessons written on its older versions. If stale or wrong lessons misled the agent, the gain over no memory would shrink for these later tasks.
Table R3. Resolution on the earlier and the later half of each repository's tasks, and the gain in points with its paired 95% interval (main comparison; tasks ordered by the number in the task identifier).
| Benchmark | Tasks of each repository | ContextGraph | No memory | Gain (95% interval) |
|---|---|---|---|---|
| Verified500 (one run) | Earlier half | 70.6% | 63.5% | +7.1 (3.2 to 11.1) |
| Verified500 (one run) | Later half | 67.7% | 59.7% | +8.1 (4.0 to 12.1) |
| Related-Lite99 (five runs) | Earlier half | 31.6% | 17.0% | +14.6 (9.4 to 20.5) |
| Related-Lite99 (five runs) | Later half | 38.8% | 26.2% | +12.6 (6.6 to 19.1) |
The gain excludes zero in each half, and the change between the halves is within its interval (+0.9 points, -4.7 to 6.8, on Verified500; -2.0 points, -10.3 to 6.3, on Related-Lite99). We think the reason is that each lesson states the condition under which it applies and the agent selects lessons for each task, so a lesson that no longer fits is not selected (Table R1). Section 4.5 and Appendix H.7 now report this analysis. These streams have at most 500 tasks, as the Limitations section states.
W2 (continued). "[...] the reported confidence intervals for improvements over the strongest baseline on each of the three main benchmarks include zero. This does not establish that the improvements are absent, but it limits the strength of the comparative claims."
Thank you for this fair and careful assessment. We have replaced the earlier statement with the paired evidence. The conservative intervals of Table 8 do not compare the methods on the same tasks. The paired analysis requested in Q3 does, and it supports the gain over no memory on all three benchmarks and the margin over the strongest baseline on DeepSWE113 (Table R4).
Table R4. Margins of ContextGraph in points, with 95% intervals. Bold marks results that exclude zero.
| Benchmark | Compared with | Margin | Conservative, unpaired (Table 8) | Paired over the same tasks | Runs as the unit (Welch test; McNemar for the single run) |
|---|---|---|---|---|---|
| Verified500 | No memory | +7.6 | ±5.9 | 4.8 to 10.6 | p < 0.001 |
| Verified500 | ReasoningBank | +2.2 | ±5.8 | -0.8 to 5.2 | p = 0.18 |
| Related-Lite99 | No memory | +13.7 | ±12.4 | 9.6 to 18.0 | p < 0.001 |
| Related-Lite99 | ExpeRepair | +3.8 | ±13.1 | 0.4 to 7.1 | p = 0.09 |
| DeepSWE113 | No memory | +11.3 | ±12.2 | 7.2 to 15.7 | p < 0.001 |
| DeepSWE113 | Supermemory | +5.9 | ±12.0 | 2.7 to 9.2 | p = 0.001 |
Pairing removes the variation in task difficulty, which dominates the unpaired intervals on benchmarks of about 100 tasks. The paired interval reflects variation across tasks with the observed runs held fixed, and the run-level test reflects variation across runs, so we call a margin supported only when both agree. Over the strongest baseline, the margin is supported under both analyses on DeepSWE113, only under the paired analysis on Related-Lite99, and under neither on Verified500. Section 4.2 now reports the paired intervals of the gain and of the margin on each benchmark, and Appendix H.6 gives both analyses with this conclusion.
Q3. "Can you report variability across the repeated runs, completed-evaluation counts for each method, and a paired analysis of the principal comparisons? How does the analysis account for tasks sharing an evolving memory?"
Thank you for this helpful request. Appendix H.6 now reports all three for every method, and Table R5 shows the principal comparisons. Each run resets memory, so the five runs of a method are independent, while tasks inside one run share a memory stream.
Table R5. Resolved / completed tasks in each run, with the mean and standard deviation across runs.
| Benchmark | Method | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Mean ± SD (%) |
|---|---|---|---|---|---|---|---|
| Related-Lite99 | ContextGraph | 37/99 | 29/97 | 35/99 | 37/99 | 35/99 | 35.1 ± 3.1 |
| Related-Lite99 | ExpeRepair | 29/96 | 29/99 | 35/96 | 31/98 | 28/97 | 31.3 ± 3.1 |
| Related-Lite99 | No memory | 22/98 | 25/98 | 17/99 | 24/99 | 17/96 | 21.4 ± 3.8 |
| DeepSWE113 | ContextGraph | 84/112 | 82/113 | 80/110 | 79/112 | 78/112 | 72.1 ± 2.1 |
| DeepSWE113 | Supermemory | 74/111 | 72/113 | 74/111 | 76/112 | 74/112 | 66.2 ± 1.5 |
| DeepSWE113 | No memory | 69/110 | 68/110 | 68/112 | 64/112 | 69/112 | 60.8 ± 2.2 |
Across all methods and benchmarks, 12,407 of 12,480 evaluations completed, and resolution uses completed evaluations as the denominator. The shared memory is handled in two ways (Table R4). The paired analysis resamples tasks and compares the two methods on the same tasks, keeping all five runs of each method. The run-level test uses only the five run rates of each method, so tasks in the same stream are never treated as independent. On Verified500, each method has one run over all 500 tasks.
Q4. "What evidence distinguishes the benefit of general debugging advice from repository-specific or recently acquired knowledge? The static-playbook ablation is useful, but it does not establish when external memory is preferable to learning general strategies in the model."
Thank you for this thoughtful question. Existing results separate the three kinds of knowledge, and Appendix H.4 now presents them together (Table R6).
Table R6. Evidence for each kind of knowledge named in the question (Related-Lite99 unless noted; no memory: 21.4%, full system: 35.1%).
| Kind of knowledge | Evidence | Where |
|---|---|---|
| General debugging advice | The static playbook gives every task the same three general lessons (reproduce before fixing, write tests before submitting, check that the tests match the requirement) and stays at 20.8%. When lessons are matched to the task, those from other repositories alone give 28.7%, a gain of 7.3 points | Tables 3 and 5 |
| Repository-specific knowledge | Adding the repository's own lessons raises 28.7% to 35.1%, a further 6.4 points | Table 5 |
| Recently acquired knowledge | Freezing the pool lowers resolution from 35.1% to 28.7% (a separate run that happens to give the same rate). On DeepSWE113, the same-repository lessons (4.6 points) were all learned during the stream | Tables 3 and 5 |
Together they show that general advice helps once it is matched to the task (7.3 points), repository-specific lessons add 6.4 points, and freezing what is learned during the stream costs 6.4 points. The split depends on the benchmark: on Verified500, lessons from other repositories give 1.4 of the 7.6 points and the repository's own lessons give 6.2 (Table 5). On the second sentence of the question, we agree that comparing external memory with learning general strategies in the model is a valuable open question, and it is now in the Limitations section.
Q5. "What access boundaries are assumed for cross-repository memory? [...] Is the intended scope restricted to repositories with shared access permissions?"
Thank you for raising this important point. Yes, the intended scope is repositories that share access permissions, and the system can enforce this boundary with a filter it already has. Every lesson keeps a link to its source trajectory and repository (Section 3.1), and selection can be restricted by source repository: the "other repositories only" setting of Table 5 is run this way. A deployment applies the same filter with the repositories that a user may read, so a lesson reaches only users who can read its source. We fully agree that removing repository names does not remove confidential content, which is why the boundary is set on the source repository and not on the text of the lesson. The Ethics statement now states this scope and this mechanism.
W1. "The paper repeatedly states its takeaways, while important operational details remain unclear. [...] Equation 3 introduces (m_j) [...]. The different memory pools and evaluation configurations are also difficult to track. Figure 3 is overcrowded and hard to read; Figure 4(b) has an overflowing '+13.7' annotation."
Thank you for these specific examples; they made the fixes easy to target. All five are in the updated version:
- The subsection titles of Section 4 are now neutral and no longer restate the takeaways, and the sentence that announced one takeaway per subsection is removed.
- The design details from Q1 and Q2 are in Section 3.3 and Appendix H.1. A configuration table in Appendix H.2 lists, for every reported result, the benchmark, initial memory, memory-building model, backbone, scaffold, number of runs, and denominator.
- Section 3.3 states the evaluated consultation policy (Q2), and a concrete memory-action trace for the Django example of Figure 2 now follows Equation 3.
- Figure 3 is now a method overview, and the organization of the graph is a separate figure in the appendix (Figure 6).
- The "+13.7" label in Figure 4(b) is now inside its panel.
Thank you again for the careful review, which made the method and its evidence much easier to assess. We hope the updated version addresses these concerns, and we would be very grateful if the reviewer would consider raising the score. We are happy to discuss any further questions.
Thank you very much for the careful review, and for asking about the details of our method. We are grateful that the research question was found "useful, practical" and that the ablations "provide useful evidence that abstracting trajectories and selecting lessons dynamically matter". Due to the page limit, we have now added the specific settings to Section 3.3 and Appendix H (changes are marked in blue), and the questions helped us improve the paper greatly. We first give the rules of the evaluated system (Q1, Q2), then the evidence and its statistical strength (W2, Q3), then Q4, Q5, and the presentation (W1).
Thank you for asking for these criteria. They are now stated in Section 3.3, with the full table in Appendix H.1 (Table R1). Lessons are admitted after every task that ends with a patch, merged at a fixed similarity threshold, and kept; stale or mistaken advice is handled when a lesson is used.
Table R1. Lesson lifecycle in the evaluated system.
The extraction prompt and the graph schema are in the anonymous repository cited in the Reproducibility statement.
Thank you for this question. Section 3.3 now states the policy. Yes, the harness triggers both consultations: every task consults memory once before the agent plans, and again after it meets an error.
On the logs, the lessons returned at each point are recorded in the agent's trajectory, so the logs show what the agent received. Whether a returned lesson was then applied is not logged, since that requires judging the agent's later edits; Figure 2 gives one worked example. Measuring this directly is a valuable suggestion, and the Limitations section now notes this gap.