acceptodds
Desk rejected at ICLR 2027

ContextGraph: Improving Coding Agents with Agent-Selected Lessons from Experience

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents repeat costly debugging mistakes even when a lesson that would avoid them was learned on an earlier task, often in another repository. We study what makes experience memory help such agents, using ContextGraph, an experience graph that distills successful and failed trajectories into strategies and warnings with the conditions under which they apply, lets the agent select them with its full context before planning and after errors, and adds every completed task back to memory. ContextGraph improves over no memory by 7.6, 13.7, and 11.3 points on SWE-bench Verified, SWE-ContextBench Related-Lite, and Datacurve DeepSWE, reaching 69.2%, 35.1%, and 72.1%, the highest rate among the evaluated memory methods on each. Its ablations show that two choices carry the gain: memory should store abstract lessons and let the agent select them for each task. On Related-Lite, raw trajectory excerpts selected by the agent and a fixed set of abstract lessons given to every task both stay at the no-memory level (20.5% and 20.8% versus 21.4%), and abstract lessons retrieved by embedding similarity leave ContextGraph only 2.9 of its 13.7 points. Updating memory after every task adds further gains, and lessons from other repositories alone carry more than half of the gain on Related-Lite and DeepSWE. The gains hold for strong agents: with Gemini 3.1 Pro, ContextGraph raises SWE-bench Verified pass@1 from 82.3% to 88.9%, above the 82.8% that five attempts reach without memory. We release the code and prompts so that others can build on these findings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, long-term memory, program repair, cross-repository transfer, experience graphs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9GP6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

ContextGraph stores lessons from coding-agent trajectories in a graph, lets agents select relevant lessons during subsequent tasks, and updates memory after each attempt. The authors evaluate the system across several benchmarks and model backbones, reporting improvements over agents without memory and existing memory systems.

Critical strengths

The paper investigates the useful, practical question of how coding agents can reuse experience across tasks. The eval includes several memory baselines and models, and the ablations provide useful evidence that abstracting trajectories and selecting lessons dynamically matter.

Critical weaknesses

W1. The presentation makes the method unnecessarily difficult to assess. The paper repeatedly states its takeaways, while important operational details remain unclear. For example, what qualifies a lesson for storage, how unreliable or contradictory lessons are handled, and whether memory consultation is enforced or merely encouraged. Equation 3 introduces (m_j), which is briefly revisited in Appendix F but contributes little beyond listing tool actions. The different memory pools and evaluation configurations are also difficult to track. Figure 3 is overcrowded and hard to read; Figure 4(b) has an overflowing “+13.7” annotation. The latter is minor, but the broader lack of clarity substantially affects my ability to assess the method.

W2. The contribution and evidence do not sufficiently justify the proposed complexity. The method combines lesson extraction, graph storage, agent-directed retrieval, and online updates. The paper does not clearly establish what substantial new insight this combination provides beyond existing memory approaches. Removing graph links retains 31.1% resolution versus 35.1% for the full system, suggesting that much of the benefit survives without the graph. Moreover, the reported confidence intervals for improvements over the strongest baseline on each of the three main benchmarks include zero. This does not establish that the improvements are absent, but it limits the strength of the comparative claims. Staleness and erroneous memory accumulation also remain untested despite their relevance to a continually growing memory.

Questions for authors

Q1. What are the actual criteria for admitting, revising, and removing lessons? How does the system handle advice that becomes stale after repository changes, conflicts with another lesson, or was inferred incorrectly from a failed attempt?

Q2. Are the memory consultations before planning and after errors enforced by the harness? Do the logs quantify whether agents read and subsequently apply the retrieved lessons?

Q3. Can you report variability across the repeated runs, completed-evaluation counts for each method, and a paired analysis of the principal comparisons? How does the analysis account for tasks sharing an evolving memory?

Q4. What evidence distinguishes the benefit of general debugging advice from repository-specific or recently acquired knowledge? The static-playbook ablation is useful, but it does not establish when external memory is preferable to learning general strategies in the model.

Q5. What access boundaries are assumed for cross-repository memory? Removing repository names does not necessarily remove confidential content, particularly when source records and online lessons retain files and commands. Is the intended scope restricted to repositories with shared access permissions?

4 comments

Thank you very much for the careful review, and for asking about the details of our method. We are grateful that the research question was found "useful, practical" and that the ablations "provide useful evidence that abstracting trajectories and selecting lessons dynamically matter". Due to the page limit, we have now added the specific settings to Section 3.3 and Appendix H (changes are marked in blue), and the questions helped us improve the paper greatly. We first give the rules of the evaluated system (Q1, Q2), then the evidence and its statistical strength (W2, Q3), then Q4, Q5, and the presentation (W1).


Q1. "What are the actual criteria for admitting, revising, and removing lessons? How does the system handle advice that becomes stale after repository changes, conflicts with another lesson, or was inferred incorrectly from a failed attempt?"

Thank you for asking for these criteria. They are now stated in Section 3.3, with the full table in Appendix H.1 (Table R1). Lessons are admitted after every task that ends with a patch, merged at a fixed similarity threshold, and kept; stale or mistaken advice is handled when a lesson is used.

Table R1. Lesson lifecycle in the evaluated system.

StageRule
AdmissionAfter each task that ends with a patch, resolved or not, one Claude Sonnet 4 call reads the issue, a digest of the agent's actions and observations, and the patch. The prompt asks for 3 to 5 one-sentence lessons of the form "When you see [symptom], do [action] so that [result]". A task with an empty patch adds none.
SafeguardThe writer is told that the patch may be wrong, and official test results never enter memory (Section 4.1).
RevisionA new lesson with cosine similarity ≥ 0.92 to a stored lesson is merged into it, and the support count of the stored lesson increases. Otherwise it is stored and linked to its source trajectory and nearest error pattern. The initial pool was merged the same way at 0.88.
RemovalLessons are kept; nothing is deleted.
Stale, conflicting, or wrongly inferred adviceHandled at use: each lesson states the condition under which it applies, the prompt tells the agent to treat lessons as advice, not rules, and the agent applies those that fit the code in front of it.

The extraction prompt and the graph schema are in the anonymous repository cited in the Reproducibility statement.


Q2. "Are the memory consultations before planning and after errors enforced by the harness? Do the logs quantify whether agents read and subsequently apply the retrieved lessons?"

Thank you for this question. Section 3.3 now states the policy. Yes, the harness triggers both consultations: every task consults memory once before the agent plans, and again after it meets an error.

On the logs, the lessons returned at each point are recorded in the agent's trajectory, so the logs show what the agent received. Whether a returned lesson was then applied is not logged, since that requires judging the agent's later edits; Figure 2 gives one worked example. Measuring this directly is a valuable suggestion, and the Limitations section now notes this gap.

W2. "The paper does not clearly establish what substantial new insight this combination provides beyond existing memory approaches. Removing graph links retains 31.1% resolution versus 35.1% for the full system, suggesting that much of the benefit survives without the graph."

Thank you for this point. Section 1 now states the insight directly. The insight is which parts of experience memory carry the gain. Existing memory systems each combine several design choices, and under one agent, backbone, and budget they range from -2.1 to +9.9 points over no memory (Table 7A). Only an ablation that removes one choice at a time can tell which choice matters (Table R2).

Table R2. Ablation on Related-Lite99: one choice removed at a time from the full system, which reaches 35.1% (five-run means from Table 3; drops in points, tested with the two analyses of Table R4; bold marks results that exclude zero).

VariantResolutionDrop from fullPaired 95% intervalRun-level Welch test
Raw trajectories, no abstraction20.5%14.610.0 to 19.2p < 0.001
Static playbook, no matching to the task20.8%14.39.7 to 19.0p < 0.001
Fixed top-k by embedding similarity24.3%10.86.5 to 15.3p = 0.001
Frozen pool, no online updates28.7%6.41.5 to 11.4p = 0.011
No graph links31.1%4.0-1.0 to 9.1p = 0.07
Planning-only consultation33.2%1.9-2.6 to 6.6p = 0.37

Four choices are supported under both analyses, and each has a clear reason. Raw trajectories lose the entire gain because an excerpt from another repository carries that repository's code, whereas an abstract lesson carries the decision and the condition under which it applies (Section 4.3). The static playbook loses it because every task receives the same three general lessons, so they are not matched to the task (Section 4.4). Fixed top-k loses most of the gain because the three nearest lessons are handed over once, whereas the agent in the full system searches memory itself and keeps the lessons that fit. Freezing the pool removes the lessons learned during the stream. The first three are the findings kindly noted in the review ("abstracting trajectories and selecting lessons dynamically matter"), and they now have paired support. The drops for graph links and for consultation after errors are not supported, and the caption of Table 3 and Appendix H.6 now say so.

On the graph links, we agree with the reviewer's reading that most of the benefit survives without them. Section 3.1 now describes them as used for retrieval and for tracing each lesson back to its source.


W2 (continued). "Staleness and erroneous memory accumulation also remain untested despite their relevance to a continually growing memory."

Thank you for highlighting this important point for a growing memory. Both effects are present in every run we report, and the gain stays at a similar size as lessons accumulate (Table R3). Every task that ends with a patch adds lessons whether or not the patch is correct, no test result enters memory, and nothing is removed, so lessons from failed attempts build up in every run. Tasks arrive in order of issue creation time (Section 3), so the later tasks of a repository are also served by lessons written on its older versions. If stale or wrong lessons misled the agent, the gain over no memory would shrink for these later tasks.

Table R3. Resolution on the earlier and the later half of each repository's tasks, and the gain in points with its paired 95% interval (main comparison; tasks ordered by the number in the task identifier).

BenchmarkTasks of each repositoryContextGraphNo memoryGain (95% interval)
Verified500 (one run)Earlier half70.6%63.5%+7.1 (3.2 to 11.1)
Verified500 (one run)Later half67.7%59.7%+8.1 (4.0 to 12.1)
Related-Lite99 (five runs)Earlier half31.6%17.0%+14.6 (9.4 to 20.5)
Related-Lite99 (five runs)Later half38.8%26.2%+12.6 (6.6 to 19.1)

The gain excludes zero in each half, and the change between the halves is within its interval (+0.9 points, -4.7 to 6.8, on Verified500; -2.0 points, -10.3 to 6.3, on Related-Lite99). We think the reason is that each lesson states the condition under which it applies and the agent selects lessons for each task, so a lesson that no longer fits is not selected (Table R1). Section 4.5 and Appendix H.7 now report this analysis. These streams have at most 500 tasks, as the Limitations section states.

W2 (continued). "[...] the reported confidence intervals for improvements over the strongest baseline on each of the three main benchmarks include zero. This does not establish that the improvements are absent, but it limits the strength of the comparative claims."

Thank you for this fair and careful assessment. We have replaced the earlier statement with the paired evidence. The conservative intervals of Table 8 do not compare the methods on the same tasks. The paired analysis requested in Q3 does, and it supports the gain over no memory on all three benchmarks and the margin over the strongest baseline on DeepSWE113 (Table R4).

Table R4. Margins of ContextGraph in points, with 95% intervals. Bold marks results that exclude zero.

BenchmarkCompared withMarginConservative, unpaired (Table 8)Paired over the same tasksRuns as the unit (Welch test; McNemar for the single run)
Verified500No memory+7.6±5.94.8 to 10.6p < 0.001
Verified500ReasoningBank+2.2±5.8-0.8 to 5.2p = 0.18
Related-Lite99No memory+13.7±12.49.6 to 18.0p < 0.001
Related-Lite99ExpeRepair+3.8±13.10.4 to 7.1p = 0.09
DeepSWE113No memory+11.3±12.27.2 to 15.7p < 0.001
DeepSWE113Supermemory+5.9±12.02.7 to 9.2p = 0.001

Pairing removes the variation in task difficulty, which dominates the unpaired intervals on benchmarks of about 100 tasks. The paired interval reflects variation across tasks with the observed runs held fixed, and the run-level test reflects variation across runs, so we call a margin supported only when both agree. Over the strongest baseline, the margin is supported under both analyses on DeepSWE113, only under the paired analysis on Related-Lite99, and under neither on Verified500. Section 4.2 now reports the paired intervals of the gain and of the margin on each benchmark, and Appendix H.6 gives both analyses with this conclusion.


Q3. "Can you report variability across the repeated runs, completed-evaluation counts for each method, and a paired analysis of the principal comparisons? How does the analysis account for tasks sharing an evolving memory?"

Thank you for this helpful request. Appendix H.6 now reports all three for every method, and Table R5 shows the principal comparisons. Each run resets memory, so the five runs of a method are independent, while tasks inside one run share a memory stream.

Table R5. Resolved / completed tasks in each run, with the mean and standard deviation across runs.

BenchmarkMethodRun 1Run 2Run 3Run 4Run 5Mean ± SD (%)
Related-Lite99ContextGraph37/9929/9735/9937/9935/9935.1 ± 3.1
Related-Lite99ExpeRepair29/9629/9935/9631/9828/9731.3 ± 3.1
Related-Lite99No memory22/9825/9817/9924/9917/9621.4 ± 3.8
DeepSWE113ContextGraph84/11282/11380/11079/11278/11272.1 ± 2.1
DeepSWE113Supermemory74/11172/11374/11176/11274/11266.2 ± 1.5
DeepSWE113No memory69/11068/11068/11264/11269/11260.8 ± 2.2

Across all methods and benchmarks, 12,407 of 12,480 evaluations completed, and resolution uses completed evaluations as the denominator. The shared memory is handled in two ways (Table R4). The paired analysis resamples tasks and compares the two methods on the same tasks, keeping all five runs of each method. The run-level test uses only the five run rates of each method, so tasks in the same stream are never treated as independent. On Verified500, each method has one run over all 500 tasks.

Q4. "What evidence distinguishes the benefit of general debugging advice from repository-specific or recently acquired knowledge? The static-playbook ablation is useful, but it does not establish when external memory is preferable to learning general strategies in the model."

Thank you for this thoughtful question. Existing results separate the three kinds of knowledge, and Appendix H.4 now presents them together (Table R6).

Table R6. Evidence for each kind of knowledge named in the question (Related-Lite99 unless noted; no memory: 21.4%, full system: 35.1%).

Kind of knowledgeEvidenceWhere
General debugging adviceThe static playbook gives every task the same three general lessons (reproduce before fixing, write tests before submitting, check that the tests match the requirement) and stays at 20.8%. When lessons are matched to the task, those from other repositories alone give 28.7%, a gain of 7.3 pointsTables 3 and 5
Repository-specific knowledgeAdding the repository's own lessons raises 28.7% to 35.1%, a further 6.4 pointsTable 5
Recently acquired knowledgeFreezing the pool lowers resolution from 35.1% to 28.7% (a separate run that happens to give the same rate). On DeepSWE113, the same-repository lessons (4.6 points) were all learned during the streamTables 3 and 5

Together they show that general advice helps once it is matched to the task (7.3 points), repository-specific lessons add 6.4 points, and freezing what is learned during the stream costs 6.4 points. The split depends on the benchmark: on Verified500, lessons from other repositories give 1.4 of the 7.6 points and the repository's own lessons give 6.2 (Table 5). On the second sentence of the question, we agree that comparing external memory with learning general strategies in the model is a valuable open question, and it is now in the Limitations section.


Q5. "What access boundaries are assumed for cross-repository memory? [...] Is the intended scope restricted to repositories with shared access permissions?"

Thank you for raising this important point. Yes, the intended scope is repositories that share access permissions, and the system can enforce this boundary with a filter it already has. Every lesson keeps a link to its source trajectory and repository (Section 3.1), and selection can be restricted by source repository: the "other repositories only" setting of Table 5 is run this way. A deployment applies the same filter with the repositories that a user may read, so a lesson reaches only users who can read its source. We fully agree that removing repository names does not remove confidential content, which is why the boundary is set on the source repository and not on the text of the lesson. The Ethics statement now states this scope and this mechanism.


W1. "The paper repeatedly states its takeaways, while important operational details remain unclear. [...] Equation 3 introduces (m_j) [...]. The different memory pools and evaluation configurations are also difficult to track. Figure 3 is overcrowded and hard to read; Figure 4(b) has an overflowing '+13.7' annotation."

Thank you for these specific examples; they made the fixes easy to target. All five are in the updated version:

  1. The subsection titles of Section 4 are now neutral and no longer restate the takeaways, and the sentence that announced one takeaway per subsection is removed.
  2. The design details from Q1 and Q2 are in Section 3.3 and Appendix H.1. A configuration table in Appendix H.2 lists, for every reported result, the benchmark, initial memory, memory-building model, backbone, scaffold, number of runs, and denominator.
  3. Section 3.3 states the evaluated consultation policy (Q2), and a concrete memory-action trace for the Django example of Figure 2 now follows Equation 3.
  4. Figure 3 is now a method overview, and the organization of the graph is a separate figure in the appendix (Figure 6).
  5. The "+13.7" label in Figure 4(b) is now inside its panel.

Thank you again for the careful review, which made the method and its evidence much easier to assess. We hope the updated version addresses these concerns, and we would be very grateful if the reviewer would consider raising the score. We are happy to discuss any further questions.

Reviewer_PKLmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ContextGraph, a method for structured accumulation and retrieval of relevant memories from past trajectory experience of solving relevant tasks and edge-cases. Specifically context graph distills experience into strategies and warnings, as abstract lessons instead of raw trajectories and stores them in a typed graph linked to their source trajectories and error patterns. It then let's the agent find relevant memories using a graph traversal.

Critical strengths
  1. The paper provides concrete, easy to implement recommendations for improving coding agent memory. Specifically:
    1. Lessons helping more than raw trajectories
    2. Better to let agents browse and fine memories
    3. Share experience across repositories
  2. Controlled comparisons help isolate the effect of each part of ContextGraph
  3. Consistent gains over mentioned baselines
  4. The code and method is reproducible
Critical weaknesses
  1. Missing Baselines
    1. ExpGraph (Feng et al., 2026) (which is mentioned in Table 1) is a relevant baseline but was not compared against, with no reason for why it was excluded. Other related baselines could be MemGovern (Wang et al., 2026) (where agents browse experience cards similar to takeaway 2)
    2. Other missing citations include MemCoder (Deng et al., 2026) which creates structured memory from past commits.
    3. Fully agent managed comparisons like Claude-code style memory or Dynamic Cheatsheet (Suzgun et al., 2026) are missing.
  2. The comparison against memory baselines (Figure 4, Table 7A) and all ablations (Table 3) use only DeepSeek V4 Pro 0813 with mini-SWE-agent. The other models (Gemini 3.1 Pro, Kimi K3, MiniMax M2.7) are compared only against no memory or simple controls, and model and scaffold change together (Gemini with live-SWE-agent, GPT-5.5 with Codex). The only other full baseline comparison (GPT-5.5 + Codex, Table 15) covers 20 tasks. It is therefore unclear whether ContextGraph's advantage over other memory methods holds beyond DeepSeek V4 Pro.
  3. Choice of lesson-writing model may conflate things. ContextGraph's lessons are written by Claude 4 Sonnet (Section 4.1). The paper does not state which model the baselines use to build their memories. If baselines use the weaker DeepSeek backbone, part of the gain may come from the stronger lesson writer rather than from the method.
Questions for authors

See Weaknesses, my main questions are about missing baselines and coverage.

2 comments

Thank you very much for the constructive review and for the concrete list of related methods, which improved the paper. We are glad that the recommendations were found "concrete, easy to implement". Changes in the updated version are marked in blue. We answer the writer question first (W3), then baseline coverage (W1) and backbone coverage (W2).


W3. "The paper does not state which model the baselines use to build their memories. If baselines use the weaker DeepSeek backbone, part of the gain may come from the stronger lesson writer rather than from the method."

Thank you for raising this important point, and we are happy to clarify. Every memory-building call uses the same model, Claude Sonnet 4, for ContextGraph and for every baseline, so the gain does not come from a stronger writer. ExpeL, ExpeRepair, ACE, and Supermemory build their initial memory with their own procedures. ReasoningBank starts from the memory that ContextGraph pre-builds, and FAISS Flat indexes ContextGraph's lessons. DeepSeek V4 Pro 0813 solves the tasks for every method.

Table R1 gives a direct check: the same Claude-written lessons reach 24.3% to 35.1% depending on how they are delivered. The reason is that a well-written lesson helps only when it reaches the task it fits. Retrieval by embedding hands over the three nearest lessons once before planning, while in the full system the agent searches memory itself and keeps the lessons whose conditions match the task (Section 4.4).

Table R1. The same Claude-written lessons under three ways of delivering them (Related-Lite99).

DeliveryResolutionWhere
Fixed top-k by embedding similarity24.3%Table 3
FAISS Flat26.8%Table 7A
ContextGraph, full system35.1%Table 3

Section 4.1 now names the memory-building model for every method, and Appendix H.2 adds a configuration table.


W1. "ExpGraph [...] is a relevant baseline but was not compared against, with no reason for why it was excluded. Other related baselines could be MemGovern [...]. Other missing citations include MemCoder [...]. Fully agent managed comparisons like Claude-code style memory or Dynamic Cheatsheet [...] are missing."

Thank you for this valuable list. We have used it to extend Section 2 (Table R2).

Table R2. Each suggested method and how it relates to the comparison.

MethodTreatmentDetails
ExpGraphPositioned in Section 2 (Table 1); not comparedIts official implementation appeared in October 2026, after the submission deadline, and it trains a retrieval controller for each benchmark.
MemGovernDiscussed in Section 2; not comparedIts memory is built from curated human repair histories, whereas the main comparison builds every memory from agent trajectories.
MemCoderNow cited in Section 2Structured memory built from past commits.
Dynamic CheatsheetNow cited in Section 2; covered through ACEACE builds on it and reaches 28.1%, versus 35.1% for ContextGraph (Table 7A).
Agent-edited memory file (Claude Code style)Not yet evaluatedThe closest evaluated variant is the static playbook, a fixed file of three general lessons in every prompt, at 20.8% (Table 3).

W2. "It is therefore unclear whether ContextGraph's advantage over other memory methods holds beyond DeepSeek V4 Pro."

Thank you for stating this boundary so clearly; we agree. The full comparison with memory baselines uses DeepSeek V4 Pro. On other backbones, ContextGraph is compared with memory methods in two settings and is the highest in both (Table R3).

Table R3. Comparisons with memory methods beyond DeepSeek V4 Pro.

SettingCompared withResultScale
GPT-5.5, Codex, LoLBenchSix memory baselines and no memoryContextGraph 3/20 at pass@1; every other method 1/20 (Table 15)20 tasks
Kimi K2.6 and MiniMax M2.7, mini-SWE-agent, Related-Lite99Mem0, random and oracle summaries, and no memory; also LangMem with Kimi K2.6ContextGraph 28.6% and 27.6%; Mem0 25.5% and 18.4%; LangMem 24.5%; no memory 18.6% and 19.4% (Figure 5)One run each

With Kimi K2.6 and MiniMax M2.7 the scaffold is the same as in the main comparison, so only the backbone changes. These runs were labeled Kimi K3 in the submission; the updated version corrects the label to Kimi K2.6, and the caption of Figure 5 now includes the LangMem result.


Thank you again for the helpful review. We hope the updated version addresses these concerns, and we would be very grateful if the reviewer would consider raising the score. We are happy to discuss any further questions.

Reviewer_PKLmOpenReview ↗

Thank you for the responses.

Thank you for the detailed response. I believe W3 is addressed.

However, W1 and W2 remain partially open, since none of the requested experiments were added.

W1: The key remaining concern is that the compared baselines do not include agent-managed memory. The static playbook is not a reasonable stand-in, since the agent cannot read or write memory in that setting. A baseline where the agent reads and writes its own memory file through tools (as in Claude Code-style memory) would test whether the fixed extraction pipeline is needed, or whether the agent can write useful abstract lessons itself.

W2: The rebuttal notes that ContextGraph is the highest method on Kimi K2.6 and MiniMax M2.7, but there it is compared only with Mem0 and LangMem, in single runs. These are not the competitive baselines: on DeepSeek V4 Pro, Mem0 reaches 26.8% on Related-Lite99 (Figure 5), well below ExpeRepair at 31.3% (Table 7A), and ContextGraph's margin over ExpeRepair is only 3.8 points. The LoLBench comparison differs by two solved tasks out of 20. The claim that ContextGraph outperforms existing memory methods therefore rests on a single backbone.

I will keep my score.

Reviewer_BnYfOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ContextGraph is a memory system for coding agents: past trajectories (from the current repo and other repos) are abstracted into lessons = (strategy or warning) + condition + action, linked to the source trajectory and error. Before planning, the agent uses the full current task context to select which lessons to inject; after an error, it queries again; after every task it distills the new trajectory back. They benchmark on SWE-bench Verified, SWE-ContextBench Related-Lite, Datacurve DeepSWE, and LoLBench Python, with DeepSeek V4 Pro and other backbones.

Critical strengths
  • This is one of the more complete ablation studies in the agent-memory space. Table 3 isolates: raw trajectories, static playbook, abstract lessons no selection, fixed top-k, agent-directed selection, no graph links, post-error search off, online updates off. Each choice has a number, and the decomposition adds up to ~13.7 points on Related-Lite.
  • The finding that abstract lessons beat raw replays (20.5% → 35.1%) and that agent-directed selection beats embedding top-k by ~10 points directly settles a debate that's been floating around Expel/ReasoningBank-style work.
  • Cross-repo learning is taken seriously: 78.4% / 86.2% / 64.1% of selected lessons come from other repos on the three benchmarks, with data-separation and near-duplicate audits (Jaccard < 1.3%) to keep train/test leakage in check.
  • Multiple backbones (DeepSeek V4 Pro, Kimi K3, MiniMax M2.7, Gemini 3.1 Pro, GPT-5.5) and multiple benchmarks mean the gain isn't a one-off.
Critical weaknesses
  • The lesson abstraction prompt and the selection policy are both LLM calls on top of the already-token-heavy agent. The paper reports accuracy gains but not the latency/cost delta; a practitioner will want to know whether the 13.7 points cost 2× tokens.
  • Initial memory is seeded from held-out task trajectories. That's legitimate for a benchmark, but it means the system is a "warm-start" memory. The cold-start question (how many tasks until ContextGraph matches the seeded performance?) isn't answered—Table 4 gives a hint but only on Related-Lite99.
  • The graph edges (links to source trajectories, error patterns) look like they should enable something beyond retrieval—like "this error usually follows that error"—but the paper only uses them as back-pointers. The graph is more a connected index than a reasoning structure; the title oversells slightly.
  • Some CIs are tight (5-run mean on 500 tasks) but a few numbers (Kimi K3, MiniMax M2.7) are explicitly single runs, which the paper flags but still mixes into the main bar chart.
Questions for authors

Q1. Agent-directed lesson selection is itself an LLM call given the full task context. Is there a risk that this selection step "leaks" the fix into the prompt—that the agent doesn't solve the bug, it recognizes the lesson? A control where selection is replaced by a random lesson matched on surface keywords would bound this. Q2. Initial memory is seeded from trajectories of held-out tasks. What is the cold-start curve—how many tasks must the agent solve from scratch before ContextGraph reaches the seeded-performance level? Table 4 hints at this but doesn't measure it directly.

2 comments

Thank you very much for the detailed and encouraging review, and for the clear support of our work. We are glad that the ablation study was found to be "one of the more complete ablation studies in the agent-memory space". We have revised the paper in response to the comments, with the changes marked in blue, and we answer each point below.


W1. "The lesson abstraction prompt and the selection policy are both LLM calls on top of the already-token-heavy agent. The paper reports accuracy gains but not the latency/cost delta; a practitioner will want to know whether the 13.7 points cost 2× tokens."

Thank you for this practical question, and we are happy to report the numbers. Memory does not cost 2× tokens; it uses fewer, most likely because the lessons shorten the agent's exploration. On Verified500, the agent loop uses 35K fewer tokens per task with ContextGraph (201K versus 236K, 14.8% fewer), and post-task distillation adds at most about 4K tokens, priced at no more than $0.023 per distilled task (Table R1). The lessons tell the agent what to check first (for example, "reproduce the failure in a minimal script" in Figure 2), so it reaches a patch in fewer steps, and this outweighs the tokens of the lessons themselves, which are already counted in the agent loop.

Table R1. Token and dollar cost of ContextGraph.

ItemCostSource
Agent loop with ContextGraph201K tokens per taskSection 4.7, Verified500
Agent loop without memory236K tokens per taskSection 4.7, Verified500
Post-task distillation, inputabout 3.0K tokens per callMeasured for this response
Post-task distillation, priceat most $0.023 per callMeasured for this response, with output counted at its 900-token cap

Distillation is one lesson-writing call after each task that ends with a patch. For this response we re-ran it on 47 Related-Lite99 trajectories with Claude Sonnet 4.5, which has the same list price as Claude Sonnet 4. We have added this table to Appendix H.3, with a pointer from Section 4.7.


W2. "Initial memory is seeded from held-out task trajectories. [...] it means the system is a 'warm-start' memory."

Q2. "[...] how many tasks must the agent solve from scratch before ContextGraph reaches the seeded-performance level?"

Thank you for this insightful observation. We fully agree that the evaluated setting is a warm start, and Section 3 now says so explicitly. The paper does not include a run from an empty pool, but three existing results show how the gain builds up. The gain grows with the amount of experience, and learning during the stream adds to the seed (Table R2).

Table R2. How the gain depends on the seed and on learning during the stream.

EvidenceResultWhere
Seed of 50 / 100 / 200 / 300 summaries20.2% / 26.3% / 28.3% / 35.1% (single runs for 50 to 200; no memory: 21.4%)Section 4.6
Freezing the pool after seeding35.1% falls to 28.7%Table 3
Lead of online updates over a static pool, by thirds of the stream-0.2, 1.0, and 3.0 tasksTable 4 (retry study, curated pool)
Same-repository lessons on DeepSWE113, all learned during the stream+4.6 pointsTable 5

We think a 50-summary seed rarely contains a lesson that fits the current task, while a larger pool does so more often. The lead of online updates grows along the stream because each finished task adds lessons for later ones. The cold-start curve is a valuable experiment, and the Limitations section now states that it is not yet measured.

W3. "The graph is more a connected index than a reasoning structure; the title oversells slightly."

Thank you for this accurate description, and we have adopted it. The links act as a connected index. Removing them still gives 31.1%, versus 35.1% with them (Table 3). The reason is that the gain comes mainly from what a lesson says and from choosing the right lesson, while the links help retrieval reach the lessons tied to the same problem or error. Section 3.1 now reads: "The links have two uses: retrieval follows them from a matched problem or error to its lessons, and each lesson keeps a link back to its source trajectory." On the title, it names lessons selected by the agent, which is where the ablation places the gain, and the text no longer presents the graph as a reasoning structure. The error-transition idea ("this error usually follows that error") is an excellent suggestion, and we have added it to the Limitations section as future work.


W4. "[...] a few numbers (Kimi K3, MiniMax M2.7) are explicitly single runs, which the paper flags but still mixes into the main bar chart."

Thank you for the careful reading. Figure 5 now marks the two single-run panels in their titles and draws their bars in a lighter shade, and its caption gives the counts (Table R3). While checking these runs, we also found a labeling error: the bars labeled Kimi K3 are runs with Kimi K2.6, and the figure, its caption, and the text are corrected.

Table R3. Single-run results in Figure 5 (resolved / completed tasks on Related-Lite99).

BackboneContextGraphNo memory
Kimi K2.6 (labeled Kimi K3 in the submission)28/9818/97
MiniMax M2.727/9819/98

Q1. "Is there a risk that this selection step 'leaks' the fix into the prompt [...]? A control where selection is replaced by a random lesson matched on surface keywords would bound this."

Thank you for this insightful question and for the suggested control. Memory does not contain the evaluated task, so a lesson cannot carry its fix. Three rules ensure this (Appendix C):

  1. Initial pools exclude every evaluated instance.
  2. Entries whose patch is close to an evaluated reference patch (token-Jaccard similarity ≥ 0.85) are removed, which affects at most 1.3% of any pool.
  3. Online lessons come only from earlier tasks in the stream (in the retry studies, also from earlier attempts at the same task), and official test results never enter memory.

The lessons are also advice on how to work, not fixes. In Figure 2, the agent uses "reproduce the failure in a minimal script" and "do not set up the full framework first", then writes the Django fix itself.

Two existing controls further show that recognizing a related case is not enough (Table R4).

Table R4. Controls for recognition (Related-Lite99, DeepSeek V4 Pro).

What the agent receivesResolutionWhere
ContextGraph, full system35.1%Table 3
Oracle: the one known related summary27.8%Figure 5
Random summaries from the same pool27.1%Figure 5

The oracle is handed the one summary known to be related to the task, which is the strongest form of the suspected leak. It gains only 0.7 points over random summaries and stays 7.3 points below the full system. We think this is because a related case alone does not tell the agent what to do in the new code, whereas a lesson states an action and the condition under which it applies. The suggested keyword-matched control is a more direct test than these two, and it is the next control we plan to run.


Thank you again for the thoughtful review and the encouraging support. We are happy to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.