acceptodds
Under review as a conference paper at ICLR 2027

MemTrace: State-Consistent Memory for Long-Horizon Coding Agents

11 views

Abstract

As coding agents take on long-horizon software evolution tasks spanning multiple files and stages, longer execution trajectories introduce two coupled challenges: (1) accumulated histories strain context budgets, and (2) repository changes can invalidate earlier execution evidence. Existing approaches address these challenges through techniques like larger context windows, compression, retrieval, or repository representations, but often fail to reconstruct a consistent task state after a context refresh or verify whether recalled evidence remains valid. Thus, we introduce MemTrace, a provenance-aware memory system that preserves execution history and aligns its reuse with the evolving task (e.g., iterative cross-file repair) and repository state. MemTrace stores history as immutable Memory Traces anchored to key information (e.g., files, symbols, tests), and organizes their execution order and dependencies in a Memory Trace Graph. When context is constrained, working memory retains only compact Memory Anchors, from which the agent can reconstruct the latest execution state and locate evidence relevant to its next action. Before restoring historical evidence, MemTrace checks its validity against the current repository state and retrieves only what the next action requires. Across three complementary long-horizon coding benchmarks, MemTrace consistently outperforms all fully evaluated baselines under the same backbone and harness, improving DeepSWE pass@1 by 21.2 points, SWE-EVO Resolved Rate by 4.4 points, and SWE-Milestone Score by 17.8 points under Codex CLI.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory, Coding Agent, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8S2zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

MemTrace addresses the problem of recovering useful execution history when a coding agent’s context is refreshed and the repository has changed. It combines immutable execution traces, a Memory Trace Graph, and a Repository State Graph to retrieve and validate historical evidence. Experiments on three benchmarks show improvements in the primary task metrics across two harnesses using one DeepSeek backbone.

Critical strengths

S1. The paper addresses a relevant practical problem. Recovering history is insufficient if that history no longer describes the repository, and explicitly checking evidence before reuse is a sensible direction.

S2. The reported gains are promising, particularly on DeepSWE. Evaluating two harnesses and ablating both graphs provides useful evidence that the system contributes beyond simply storing more history.

Critical weaknesses

W1. The contribution relative to closely related approaches remains unclear. Ledger and MAGE are discussed but not evaluated, while EA-Graph and ContextSniper deserve consideration given their treatment of evidence freshness and repository-synchronized memory. Comparisons against Native, Observation Masking, and RepoGraph do not sufficiently establish what the proposed combination adds over these closer approaches.

W2. The evaluation does not fully establish the claimed mechanism or its generality. All experiments use deepseek-v4-flash-0731-high. The graph ablations are useful, but they do not isolate freshness validation, which is central to the paper’s motivation. The diagnostic counts measure file changes and policy-triggered invalidations rather than independently verified stale-evidence errors. Furthermore, the main performance results lack uncertainty estimates or reported repeated-run variability, and the runtime trade-offs leave the benefit under matched resource budgets unclear.

Questions for authors

Q1. What is the specific methodological distinction from Ledger and EA-Graph, and how does MemTrace compare with a simpler system that combines recoverable history with revision-based invalidation?

Q2. Can you isolate the contribution of validity checking while keeping storage and retrieval unchanged? How often does it prevent reuse of evidence that is actually incorrect or inapplicable?

Q3. Do you have results with another capable backbone? Two harnesses provide useful coverage, but do not establish that the findings generalize across models.

Q4. How many independent runs were performed per configuration, and can you provide uncertainty estimates for the main results and ablations?

The abstract should also qualify “consistently outperforms” as referring to the primary benchmark metrics. Table 2 contains lower Fix and Apply scores, which the main text appropriately acknowledges.

4 comments

Dear Reviewer 8S2z,

Thank you for your thoughtful and constructive review. We sincerely appreciate your recognition of the relevance of the problem and of the reported gains, and we take your concerns about closer baselines, the validation mechanism, and generality seriously. We hope the responses below address them, and we welcome any further questions or suggestions.

Terms. A trace is one stored unit of completed work. The Memory Trace Graph (MTG) links traces; the Repository State Graph (RSG) describes current files, symbols, and tests. Validation checks that the code a trace describes is unchanged before reuse. A trace that fails is closed: kept as history, not counted as current evidence.

Setup. Each configuration was run three times; tables report the mean ± standard deviation. Backbones: DS4 (deepseek-v4-flash-0731) and Qwen (Qwen3.8-Flash). Harnesses: Codex CLI and mini-swe-agent. All methods use a 200K-token context window and a 12-hour limit per task. Metrics: DeepSWE pass@1 (%), SWE-EVO Resolved Rate (%), and SWE-Milestone Score. SWE-EVO scores are lower than in the submission (15–17 points for Native): agents sometimes read the benchmark's test files during a run, and the new runs prevent this.


▶ Q1 (W1). Distinction from Ledger and EA-Graph; comparison with a simpler system

Thank you for this helpful question.

  • Ledger attaches change counters to outputs still visible in the context, to avoid redundant actions. It keeps no recoverable history, so nothing can be restored once an output leaves the context. A counter shows that something changed, not which later work replaced or re-verified it; the MTG records this explicitly in contrast,
  • EA-Graph maintains what remains provable; MemTrace reconstructs what remains actionable. EA-Graph maintains the validity of verification claims by anchoring them to fine-grained artifacts and detecting changes in their supporting evidence. In contrast, MemTrace addresses execution-state recovery rather than claim maintenance. It preserves heterogeneous execution evidence and its temporal, dependency, and correction relations, aligns candidate traces with the current repository and execution frontier, and selectively restores validated evidence required for continued execution. Thus, validation in EA-Graph is the primary objective, whereas in MemTrace it is one component of state-consistent memory reconstruction.

All methods. Simple Memory is the simpler system in the question: an append-only history, keyword retrieval, and a rule that accepts only records from the current workspace revision. MAGE and ContextSniper were adapted to coding tasks.

HarnessMethodDeepSWE (DS4)DeepSWE (Qwen)SWE-EVO (DS4)SWE-EVO (Qwen)SWE-Milestone (DS4)SWE-Milestone (Qwen)
Codex CLINative35.4 ± 1.844.2 ± 1.530.4 ± 2.224.6 ± 2.528.6 ± 1.627.1 ± 9.6
Observation Masking31.3 ± 2.840.4 ± 1.828.3 ± 2.223.2 ± 1.324.4 ± 7.822.2 ± 1.1
RepoGraph31.4 ± 3.4†52.0 ± 1.7†27.5 ± 2.525.4 ± 4.545.2 ± 5.1†42.3 ± 22.7†
Simple Memory36.9 ± 0.546.0 ± 2.731.9 ± 3.326.1 ± 3.827.1 ± 6.526.6 ± 2.2
MAGE-SWE-adapted32.2 ± 2.740.7 ± 1.528.3 ± 2.221.0 ± 1.321.9 ± 6.420.7 ± 10.7
ContextSniper-SWE-adapted36.0 ± 3.745.4 ± 2.229.0 ± 1.326.1 ± 2.226.1 ± 2.827.6 ± 3.7
Full MemTrace56.0 ± 1.058.1 ± 1.942.7 ± 3.340.6 ± 3.346.4 ± 2.344.3 ± 2.3
mini-swe-agentNative48.1 ± 2.743.1 ± 4.037.0 ± 2.226.8 ± 1.334.5 ± 3.132.7 ± 13.2
Observation Masking44.5 ± 2.239.8 ± 3.931.9 ± 1.324.6 ± 5.024.9 ± 1.722.6 ± 6.7
RepoGraph30.4 ± 4.5†51.0 ± 4.5†34.8 ± 3.827.5 ± 3.330.8 ± 10.9†33.6 ± 15.5†
Simple Memory50.1 ± 1.844.8 ± 4.238.4 ± 1.328.3 ± 2.233.6 ± 0.832.0 ± 1.6
MAGE-SWE-adapted45.1 ± 3.241.0 ± 2.735.5 ± 2.526.1 ± 2.227.1 ± 7.426.2 ± 9.6
ContextSniper-SWE-adapted49.6 ± 3.144.5 ± 4.137.7 ± 4.529.0 ± 2.531.4 ± 8.635.1 ± 7.8
Full MemTrace61.7 ± 1.857.8 ± 2.346.4 ± 3.337.7 ± 2.548.3 ± 3.947.2 ± 5.4

† Python subset (RepoGraph supports Python only): 34 DeepSWE tasks, one SWE-Milestone itinerary.

Full MemTrace is 8.0–19.1 points above the best baseline evaluated on the full task set, in all 12 settings. Simple Memory rejects every earlier record once any file is edited and has no link from a repair to the test that verified it. MAGE's rollback was rarely triggered. Ledger and EA-Graph were not run. These results will be added to the paper.

▶ Q2 (W2). Isolating validation; how often it prevents reuse of inapplicable evidence

Thank you for this important question.

Ablation (Codex CLI). Full − Validation keeps storage, both graphs, and retrieval unchanged and turns off only validation. "− X" is Full MemTrace without X; "Store + flat" is Trace Store Only in Table 3.

BenchmarkBackboneStore + flat− RSG− MTG− ValidationFull MemTraceGain
DeepSWEDS436.9 ± 2.836.6 ± 0.539.8 ± 2.338.9 ± 2.356.0 ± 1.0+17.1
DeepSWEQwen45.7 ± 1.047.2 ± 2.648.1 ± 3.149.3 ± 1.458.1 ± 1.9+8.8
SWE-EVODS431.9 ± 2.537.0 ± 3.834.1 ± 3.333.3 ± 3.342.7 ± 3.3+9.4
SWE-EVOQwen26.1 ± 3.831.2 ± 5.029.0 ± 2.528.3 ± 2.240.6 ± 3.3+12.3
SWE-MilestoneDS435.6 ± 2.739.1 ± 2.040.2 ± 5.441.3 ± 6.246.4 ± 2.3+5.1
SWE-MilestoneQwen34.3 ± 2.338.4 ± 1.638.9 ± 4.939.5 ± 0.944.3 ± 2.3+4.8

Removing validation lowers the result in all six settings, by 4.8–17.1 points.

How often. In the logs of 268 Full − Validation runs, 63 recalls asked about the current workspace state, and 12 of them (19.0%) returned closed evidence whose file had in fact changed (10 of 26, 38.5%, among scored runs that were not later rerun). With validation on, all 138 recalls of closed evidence were labeled as history, so none was reused as current.

A verified case. In a run on SWE-EVO iterative__dvc_1.10.2_1.11.0, eight tests passed and the agent then edited dvc/repo/experiments/__init__.py. Replaying the same tests gave 8/8 passes before the edit, 0/8 after it, and 8/8 after restoring only that file, so reusing the earlier result would have certified failing code. Closed evidence is not always wrong: 50 of 52 closed passing tests passed again when rerun.


▶ Q3 (W2). A second backbone

Thank you for this suggestion; we added Qwen3.8-Flash as a second backbone. With Qwen (Qwen columns of the table in Q1), Full MemTrace is 10.9–17.2 points above Native in all six settings, similar to DS4 (9.4–20.6).


▶ Q4 (W2). Number of runs, uncertainty, and resource use

Thank you for raising this point.

  • Runs. Every configuration has three runs (one in the submission); all tables report the mean ± standard deviation.
  • Consistency. Full MemTrace is above Native in all 12 benchmark–backbone–harness settings, with a mean gain of 14.6 points (95% confidence interval 12.6–16.5).
  • Resources. Full MemTrace uses 1.2–1.7 times the tokens of Native. Tokens alone do not explain the gains: Full − Validation uses 0.81–1.15 times the tokens of Full MemTrace and scores 4.8–17.1 points lower.

▶ Abstract. Qualifying "consistently outperforms"

Thank you for this suggestion; we agree. The revised abstract will state that MemTrace outperforms the evaluated baselines on the primary metric of each benchmark (DeepSWE pass@1, SWE-EVO Resolved Rate, SWE-Milestone Score), not on every secondary metric such as SWE-EVO Fix and Apply. The abstract and Tables 2–3 will be updated with the three-run results, and the method section will define each term at first use.

Thank you again for your time and constructive feedback.

Thank you for the additional experiments. The validation ablation, second backbone, and repeated runs meaningfully address several of my concerns and strengthen the empirical evidence.

However, substantial issues remain with the paper’s clarity and presentation. Figures 1 and 2 are overcrowded, with small labels, numerous icons, and competing arrows that make the core mechanism difficult to follow. These figures should help explain the method, but currently add to the reader’s effort. The method description also needs to accurately reflect the implemented validity checks.

Beyond presentation, the distinctive contribution over simpler memory systems with file-level invalidation remains insufficiently established. While the additional results are valuable, I do not think they resolve these concerns sufficiently for acceptance. I am therefore maintaining my rating of 2: Weak rejection

Dear Reviewer 8S2z,

Thank you for the follow-up comment. We agree that the original figures made the mechanism harder to follow. We have uploaded a revised manuscript through the OpenReview Revision option.

The revision makes three targeted changes.

  1. Figure 1 (Introduction, p. 2) follows one execution history through refresh and compares Native, Simple Memory, and MemTrace.

  2. Figure 2 (Section 3, p. 4) presents three stages: sealing traces, MTG-RSG candidate localization, and the applicability check before reuse.

  3. Section 3.3 now matches the implementation. Code observations and changes remain current when their bound entities are unaffected. Tool, test, test-failure, verifier, and requirement-review records close after every workspace revision. There is no per-test dependency tracking. Closed records remain historical, and a closed test result requires fresh verification before current use.

We also want to clarify the remaining distinction from a simpler memory system. The Simple Memory baseline is an append-only execution log with BM25 retrieval and a global workspace-revision filter. It has no execution frontier, milestone relations, MTG-RSG traversal, or type-specific applicability decision. Its global filter is stricter than file-level invalidation and discards unchanged-code records, but does not explain which later trace superseded them. A file-level filter still decides eligibility only, without frontier reconstruction or relation-aware retrieval. MemTrace reconstructs the unfinished frontier, connects repairs and tests through MTG relations, localizes current entities through the RSG, and restores candidates only after type-specific checks. The contribution is this recovery-and-validation mechanism, rather than invalidation alone.

The results support this separation. Under Codex CLI with DS4, Full MemTrace improves over Simple Memory by 19.1 points on DeepSWE (56.0 vs. 36.9), 10.8 on SWE-EVO (42.7 vs. 31.9), and 19.3 on SWE-Milestone (46.4 vs. 27.1). Across six Codex CLI settings, the gain is 10.8–19.3 points. Removing only validation, with storage, both graphs, retrieval, tasks, and budget fixed, lowers scores by 4.8–17.1 points.

The logs provide a direct diagnostic. Without validation, 12/63 recalls about the current workspace (19.0%) returned closed evidence whose bound file had changed. With validation, all 138 closed-evidence recalls carried a historical label. In one SWE-EVO run, eight tests passed, then the agent edited dvc/repo/experiments/__init__.py; replaying them gave 0/8 passes after the edit and 8/8 after restoring only that file. Reusing the earlier result would have certified failing code.

We understand the remaining concern about how much of the gain is distinctive to state-consistent recovery. The revised figures and implementation description make the boundary explicit, while the preceding ablation and Simple Memory results provide quantitative evidence. Although this does not exhaust every file-level baseline, it isolates the added recovery mechanism over the implemented simple baseline. We thank you again for helping us improve the paper.

Reviewer_aDg9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemTrace is a memory layer for coding agents that work across context refreshes while the repository keeps changing. Completed units of work are sealed into immutable Memory Traces, each bound to the repository state and to the files, symbols and tests it refers to. A Memory Trace Graph (MTG) records depend, verify, correct and supersede relations among traces, and a Repository State Graph (RSG) records current files, symbols, tests and their structural relations. When context is tight, trace bodies are offloaded and replaced by anchors and short synopses. After a refresh, MemTrace reconstructs the execution frontier, finds candidate traces through both graphs, and restores only those that pass an applicability check (aligned with the current repository and not superseded). With deepseek-v4-flash under Codex CLI and mini-swe-agent, MemTrace outperforms Native, Observation Masking and RepoGraph on DeepSWE (for example 35.4 to 56.6 pass@1 under Codex CLI), SWE-EVO and SWE-Milestone. An ablation removes the MTG and the RSG, and appendix analyses measure how often a trace's bound files change after it is sealed.

Critical strengths
  1. The problem is framed concretely and backed by measurement. Among 9,226 file-bound traces, 57.0% see a later change to a bound file, and the estimated probability that bound files remain unchanged falls to about 0.50 after 1,000 event steps. This is a good argument for checking evidence before reusing it, and the Fig. 1 example (an old test failure retrieved after it was already fixed) makes the failure mode easy to see.

  2. The DeepSWE gains are large and consistent: 40 to 64 of 113 tasks under Codex CLI and 55 to 75 under mini-swe-agent, with matched backbone and harness, and the gains hold on both the Python and non-Python subsets. The paper also reports metrics where MemTrace does not win (SWE-EVO Fix Rate under Codex CLI, Apply under mini-swe-agent).

Critical weaknesses
  1. The core mechanism is not isolated. The paper's thesis is that validation before restoration (Eq. 2) is what matters, yet the ablation (Table 3) only removes the MTG and the RSG. There is no arm that keeps the store and both graphs and disables applies(), so the gains could come from persistent storage and graph-routed retrieval alone. The ablation partly cuts against the narrative: on SWE-EVO, Trace Store Only already reaches 65.2 (Native 47.8), and removing the RSG changes nothing (69.6), so most of the SWE-EVO gain comes from persistence. The gains are also not tied to refresh events: results are not broken down by the number of context refreshes, although the paper already logs those counts (Appendix A.4). In addition, Appendix A.3 says test results and similar records are invalidated on every workspace revision regardless of which files changed, whereas Sec. 3.3 describes checking a test's own dependencies. If the evaluated system uses the A.3 rule, the validation is much simpler than described.

  2. The evidence is thin and one baseline looks misconfigured. Everything uses one backbone and one run, with no variance. On SWE-EVO a single task is 2.17 pp, and the SWE-Milestone Python subset has 12 milestones, so several margins amount to 1 to 4 tasks. Under mini-swe-agent, Observation Masking scores 26.6 on DeepSWE and 23.9 on SWE-EVO, about half of Native (48.7, 52.2). Lindenbauer et al. find masking competitive with LLM summarization, so this points to a configuration problem that needs to be explained. Token usage is not reported, and MemTrace is slower under Codex CLI (54.17 vs 42.02 minutes per task on DeepSWE, 82.05 vs 30.00 on SWE-EVO), so a cost-matched comparison is missing.

Questions for authors
  1. (Could change my rating) An ablation that keeps the Trace Store, MTG and RSG but restores retrieved candidates without the applies() check, under the same budget; and results broken down by the number of context refreshes or by the horizon groups in Appendix A.4.

  2. (Clarification) Which invalidation rule does the evaluated system use for test evidence: the per-dependency check of Sec. 3.3, or the A.3 rule that closes TEST_RESULT and TEST_FAILURE records on any revision? Similarly, Appendix A.2 says the RSG is symbol-level only for Python and file-level elsewhere; how does that affect the non-Python results?

  3. (Clarification) How are supersede and correct edges created (declared by the agent, by heuristics, or by an LLM), and how accurate are they? Eq. 2 depends on them.

  4. (Clarification) What window was used for Observation Masking, and why does it fall to about half of Native under mini-swe-agent?

  5. (Could change my rating) Token usage per task for every method, and repeated runs with paired tests.

  6. (Related work) MAGMA (Jiang et al., 2026) appears in the bibliography but is not discussed. How does the MTG/RSG design differ from its multi-graph memory?

6 comments

Dear Reviewer aDg9,

Thank you for your detailed and constructive review. We sincerely appreciate your recognition of the concrete problem framing and of the DeepSWE results, and we take your concerns about isolating validation, the Observation Masking baseline, and the strength of the evidence seriously. We hope the responses below address them, and we welcome any further questions or suggestions.

Setup. Each configuration was run three times; tables report the mean ± standard deviation. Backbones: DS4 (deepseek-v4-flash-0731) and Qwen (Qwen3.8-Flash). Harnesses: Codex CLI and mini-swe-agent. All methods use a 200K-token context window and a 12-hour limit per task. Metrics: pass@1 (%) on DeepSWE (113 tasks), Resolved Rate (%) on SWE-EVO (46 tasks), and Score on SWE-Milestone (7 itineraries, 98 graded milestones).

SWE-EVO. After submission, we found that agents sometimes read the benchmark's test files during SWE-EVO runs, which could leak the expected behavior. The new runs prevent this, so SWE-EVO scores are lower than in the submission (Native: 47.8 → 30.4 under Codex CLI, 52.2 → 37.0 under mini-swe-agent).

Main results. The table below will replace Table 2. Simple Memory, MAGE-SWE-adapted, and ContextSniper-SWE-adapted are new baselines.

HarnessMethodDeepSWE (DS4)DeepSWE (Qwen)SWE-EVO (DS4)SWE-EVO (Qwen)SWE-Milestone (DS4)SWE-Milestone (Qwen)
Codex CLINative35.4 ± 1.844.2 ± 1.530.4 ± 2.224.6 ± 2.528.6 ± 1.627.1 ± 9.6
Observation Masking31.3 ± 2.840.4 ± 1.828.3 ± 2.223.2 ± 1.324.4 ± 7.822.2 ± 1.1
RepoGraph31.4 ± 3.4†52.0 ± 1.7†27.5 ± 2.525.4 ± 4.545.2 ± 5.1†42.3 ± 22.7†
Simple Memory36.9 ± 0.546.0 ± 2.731.9 ± 3.326.1 ± 3.827.1 ± 6.526.6 ± 2.2
MAGE-SWE-adapted32.2 ± 2.740.7 ± 1.528.3 ± 2.221.0 ± 1.321.9 ± 6.420.7 ± 10.7
ContextSniper-SWE-adapted36.0 ± 3.745.4 ± 2.229.0 ± 1.326.1 ± 2.226.1 ± 2.827.6 ± 3.7
Full MemTrace56.0 ± 1.058.1 ± 1.942.7 ± 3.340.6 ± 3.346.4 ± 2.344.3 ± 2.3
mini-swe-agentNative48.1 ± 2.743.1 ± 4.037.0 ± 2.226.8 ± 1.334.5 ± 3.132.7 ± 13.2
Observation Masking44.5 ± 2.239.8 ± 3.931.9 ± 1.324.6 ± 5.024.9 ± 1.722.6 ± 6.7
RepoGraph30.4 ± 4.5†51.0 ± 4.5†34.8 ± 3.827.5 ± 3.330.8 ± 10.9†33.6 ± 15.5†
Simple Memory50.1 ± 1.844.8 ± 4.238.4 ± 1.328.3 ± 2.233.6 ± 0.832.0 ± 1.6
MAGE-SWE-adapted45.1 ± 3.241.0 ± 2.735.5 ± 2.526.1 ± 2.227.1 ± 7.426.2 ± 9.6
ContextSniper-SWE-adapted49.6 ± 3.144.5 ± 4.137.7 ± 4.529.0 ± 2.531.4 ± 8.635.1 ± 7.8
Full MemTrace61.7 ± 1.857.8 ± 2.346.4 ± 3.337.7 ± 2.548.3 ± 3.947.2 ± 5.4

† Python subset (RepoGraph supports Python only): 34 DeepSWE tasks, one SWE-Milestone itinerary.

Full MemTrace has the highest primary metric in all 12 settings, 8.0–19.1 points above the best full-set baseline.

▶ Q1 (W1). Ablation without applies(); breakdown by horizon group

Thank you for this valuable suggestion; we ran both analyses.

1. Ablation. Full − Validation keeps the Trace Store, MTG, RSG, and retrieval, and restores retrieved candidates without the applies() check. All arms use Codex CLI under the same budget. "− X" is Full MemTrace without X; "Store + flat" is Trace Store Only in Table 3.

BenchmarkBackboneNativeStore + flat− RSG− MTG− ValidationFull MemTrace
DeepSWEDS435.4 ± 1.836.9 ± 2.836.6 ± 0.539.8 ± 2.338.9 ± 2.356.0 ± 1.0
DeepSWEQwen44.2 ± 1.545.7 ± 1.047.2 ± 2.648.1 ± 3.149.3 ± 1.458.1 ± 1.9
SWE-EVODS430.4 ± 2.231.9 ± 2.537.0 ± 3.834.1 ± 3.333.3 ± 3.342.7 ± 3.3
SWE-EVOQwen24.6 ± 2.526.1 ± 3.831.2 ± 5.029.0 ± 2.528.3 ± 2.240.6 ± 3.3
SWE-MilestoneDS428.6 ± 1.635.6 ± 2.739.1 ± 2.040.2 ± 5.441.3 ± 6.246.4 ± 2.3
SWE-MilestoneQwen27.1 ± 9.634.3 ± 2.338.4 ± 1.638.9 ± 4.939.5 ± 0.944.3 ± 2.3

Removing only the applies() check lowers the result in all six settings, by 4.8–17.1 points.

2. Persistence alone does not explain the gains, including on SWE-EVO. Store + flat is 1.5 points above Native on DeepSWE and SWE-EVO and about 7 points above on SWE-Milestone, while Full MemTrace is 12.3–20.6 points above Native. On SWE-EVO, removing the RSG costs 5.7 (DS4) and 9.4 (Qwen) points. On DeepSWE and SWE-EVO, removing any single component loses about half or more of the gain, so the components work together.

3. Horizon groups of Appendix A.4 (DeepSWE, Codex CLI, mean of the two backbones).

Horizon group (tasks)NativeFull − ValidationFull MemTraceFull MemTrace − NativeGain from validation
Short (37)56%57%66%+10+9
Medium (38)40%46%59%+19+13
Long (38)24%30%47%+23+17
Overall (113)39.8%44.1%57.1%+17.3+13.0

The gain over Native grows from 10 points on short tasks to 23 on long tasks, and the gain from validation from 9 to 17.

4. Context refreshes. Full MemTrace under Codex CLI averages 2.7–3.0 refresh events per DeepSWE task and 3.6–6.3 per SWE-EVO task. In Appendix A.4, median refreshes rise from 2 (Short) to 3 (Long) and repository revisions from 21 to 41, so the larger gains on long tasks coincide with more refreshes and code changes.


▶ Q2. Rule for test evidence; effect of the file-level RSG on non-Python repositories

Thank you for the careful reading.

1. The evaluated system uses the A.3 rule. The implementation follows Appendix A.3: at each workspace revision, earlier TEST_RESULT and TEST_FAILURE records lose their current validity, regardless of which files changed. They remain available as history, but fresh verification is required before they can support current acceptance. We do not implement per-test dependency tracking and will correct Section 3.3 accordingly. Section 3.3 will be revised, and the ablation in Q1 measures the implemented rule. Test and tool results are 9.7% of records. Code observations and changes (84.2%) are closed only when their bound file changes: one revision closes 5.7% of the current code evidence on average, and none where the file is unchanged (88,724 checks).

2. File-level links on non-Python repositories. RSG navigation granularity and evidence invalidation are separate. File-level representations still associate execution traces with versioned files, allowing recovery and revision checks without symbol-level parsing. With file-level links, an edit to one symbol also closes evidence bound to other symbols in the same file. This adds file reads and never keeps outdated evidence current. In the execution logs, 40 of 386 file-level supersede edges (10.4%) connect traces about different symbols in the same file.

The gains remain: on SWE-Milestone (DS4, Codex CLI), Full MemTrace improves over Native by 15.4 points on the Python itinerary (scikit-learn, 30.7 → 46.1) and by 18.2 points on average on the six non-Python itineraries (+5.7 to +47.5). This breakdown will be added to the paper.

▶ Q3. Creation and accuracy of supersede and correct edges

Thank you for this question. Both are created by fixed runtime rules from recorded events, not declared by the agent or judged by an LLM.

  • supersede: when a code change is recorded, the runtime links the latest earlier implementation trace bound to the same file or symbol to the new trace.
  • correct: added only when three events occur in order: a milestone verification fails, the code changes, and the verification then passes.

The edges were audited against the stored events, file hashes, and bindings:

  • 645 of 645 supersede edges link to a recorded code change and an earlier trace bound to the same code.
  • The replaced code evidence was closed in 624 of 624 cases.
  • The two traces have different file contents in 631 of 645 edges (97.8%).
  • 346 of 386 file-level edges (89.6%) connect traces about the same symbol; the other 40 are the over-closing case in Q2.

The edges were not labeled by hand. No correct edge occurred in the audited runs, so no accuracy is reported for it.


▶ Q4 (W2). Observation Masking: window, and the gap to Native under mini-swe-agent

Thank you for pointing this out.

1. Window. The ten most recent tool observations are kept in full, and older ones are replaced by short placeholders (LastNObservations, n = 10). Reasoning and actions stay in the context.

2. The gap came from the time limits of the submission. The submission allowed 3 hours per DeepSWE task and 4 hours per SWE-EVO task. Under mini-swe-agent, Observation Masking makes 1.6–2.1 times the model calls of Native per task, so most of its runs reached the limit before finishing. We also removed a redundant second masking pass in our adapter; it did not change which ten observations were kept. With the 12-hour limit and three runs (main table), Observation Masking under mini-swe-agent with DS4 reaches 44.5 on DeepSWE (Native 48.1) and 31.9 on SWE-EVO (Native 37.0). Across all 12 settings, it is 1.4–10.1 points below Native.

3. Why Observation Masking does not suit long-horizon tasks. Masking has no separate memory: an observation outside the ten most recent is recovered only by running the tool again. A file read at step 5 is masked after ten newer observations and is read again when needed at step 30. On long tasks this repeats many times, so Observation Masking uses 1.3–1.9 times the tokens of Native (Q5) and still scores lower. Reasoning and actions are not masked, so the context still fills up on the longest tasks. MemTrace stores each trace outside the context and recalls it by address.

These details will be added to the paper.

Reviewer_AJCdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MemTrace, a state-consistent memory system for long-horizon coding agents that addresses task-state drift caused by context refreshes and evolving repository states. MemTrace stores immutable Memory Traces, organizes execution dependencies through a Memory Trace Graph (MTG), and connects historical evidence to current repository entities using a Repository State Graph (RSG). Experiments across DeepSWE, SWE-EVO, and SWE-Milestone report improvements over the evaluated baselines under two agent harnesses.

Critical strengths
  • The paper is trying to address an important practical challenge in long-horizon coding agents, where context refreshes and repository changes can lead to stale evidence reuse and inconsistent task-state recovery.
  • The integration of execution-history provenance with evolving repository states offers a well-motivated and relatively novel approach to maintaining memory consistency beyond conventional context compression and retrieval.
  • The experimental evaluation covers three long-horizon coding benchmarks and two agent harnesses, demonstrating substantial performance improvements and providing useful evidence through component ablations.
Critical weaknesses
  • This paper frames state-memory misalignment as a fundamental challenge in long-horizon coding agents, but its proposed solution largely consists of harness-level context checkpointing, structured history retrieval, and revision-based evidence filtering (Section 3 and Algorithm 1). The limited algorithmic novelty beyond existing memory management techniques raises concerns about the significance and generality of its research contribution.
  • This paper proposes a state-consistent restoration mechanism based on the aligned and superseded predicates (Eq. 2), but does not sufficiently specify how test dependencies and MTG supersession relations are constructed and validated, leaving the reliability of evidence applicability checks unclear.
  • This paper attributes its performance improvements to state-consistent memory restoration, but the component ablations in Table 3 do not isolate the contribution of evidence validation from structured retrieval and persistent memory storage. Moreover, the state-memory alignment analysis in Figure 4 measures repository changes and policy-triggered invalidations rather than directly verifying stale evidence identification, leaving the core mechanism insufficiently supported by experimental evidence.
Questions for authors
  1. What are the key technical distinctions between MemTrace and simpler harness-level memory management approaches that combine persistent execution logs, structured retrieval, and revision-based invalidation? Which aspects of the proposed mechanism cannot be achieved through these existing techniques?
  2. How are the aligned and superseded predicates operationally implemented, particularly for tracking test dependencies and identifying supersession relations? How does MemTrace ensure the reliability of these applicability decisions?
  3. To what extent do the performance improvements come specifically from state-consistent evidence validation rather than persistent storage and structured retrieval? Is there direct evidence that MemTrace correctly identifies stale evidence and prevents its reuse during context recovery?
3 comments

Dear Reviewer AJCd,

Thank you for your careful and constructive review. We sincerely appreciate your recognition of the practical importance of the problem and of our approach, and we take your concerns about novelty, the validity checks, and the evidence for validation seriously. We hope the responses below address them, and we welcome any further questions or suggestions.

Setup. Each configuration was run three times; tables report the mean ± standard deviation. Backbones: DS4 (deepseek-v4-flash-0731) and Qwen (Qwen3.8-Flash). All methods use a 200K-token context window and a 12-hour limit per task. Metrics: DeepSWE pass@1 (%), SWE-EVO Resolved Rate (%), and SWE-Milestone Score. SWE-EVO scores are lower than in the submission (15–17 points for Native): agents sometimes read the benchmark's test files during a run, and the new runs prevent this.


▶ Q1 (W1). Distinction from persistent logs, structured retrieval, and revision-based invalidation

Thank you for this important question. We implemented this combination as a new baseline, Simple Memory: an append-only execution log, keyword retrieval (BM25), and a filter that accepts only records written at the current workspace revision. MemTrace adds three mechanisms.

  1. Task progress is restored. MemTrace records execution milestones, their completion criteria, and the traces that satisfy them. After a context refresh, Memory Anchors restore the goal, the current milestone, and the unfinished work. Simple Memory returns separate log entries.
  2. Retrieval follows recorded links. The MTG records which trace depends on, replaces, or verifies another; the RSG records current files, symbols, and tests; each trace is bound to its code. A recall starting at test_cache follows the code link to CacheManager.invalidate and returns the repair and test traces for that code. Keyword retrieval returns only matching text.
  3. Each piece of evidence has its own validity rule. Simple Memory rejects every earlier record once any file changes. MemTrace closes only the evidence affected by the edit (Q2): after the agent reads the cache code, passes its tests, and edits a documentation file, MemTrace keeps the code observation and requires only a test rerun. In Simple Memory's DeepSWE runs (DS4, Codex CLI), this filter rejected 60,893 retrieved candidates and kept 5,881.

Codex CLI results:

BenchmarkBackboneNativeSimple MemoryFull MemTraceFull MemTrace − Simple Memory
DeepSWEDS435.4 ± 1.836.9 ± 0.556.0 ± 1.0+19.1
DeepSWEQwen44.2 ± 1.546.0 ± 2.758.1 ± 1.9+12.1
SWE-EVODS430.4 ± 2.231.9 ± 3.342.7 ± 3.3+10.8
SWE-EVOQwen24.6 ± 2.526.1 ± 3.840.6 ± 3.3+14.5

Simple Memory is 1.5–1.8 points above Native; Full MemTrace is 12.3–20.6 points above. Across all 12 benchmark–backbone–harness settings, Full MemTrace is 9.4–20.6 points above Native and 13.1–24.7 points above Observation Masking, which supports the generality of the contribution. Simple Memory will be added to the paper.

▶ Q2 (W2). Implementation and reliability of aligned and superseded

Thank you for raising this point. The runtime computes both predicates from saved events, file contents, and trace-to-code bindings. The model never decides whether a record is current.

aligned. A workspace revision is a hash of the file contents. When it changes, one rule per evidence type applies. Closed means that the record no longer counts as current evidence; it stays stored and is recalled only with a "historical" label.

Evidence typeShareClosed when
Code observations and code changes84.2%the bound file changes
Test, tool, and verifier results9.7%any workspace file changes
Plans, design decisions, and user constraints6.1%never by a code change alone

Test dependencies. MemTrace does not track which files a test depends on: any edit closes a test result, and a fresh run is required before it can satisfy a milestone completion criterion (Appendix A.3). Section 3.3 suggested a per-test dependency check and will be revised.

superseded. When code changes, the runtime adds a supersede edge from the latest earlier implementation trace bound to the same file or symbol to the new trace. A trace counts as superseded only when such an edge exists; a later timestamp alone is not enough.

Reliability. The rules are conservative: closing a correct record costs one extra file read or test run, whereas keeping an outdated record current could mislead the agent. The execution logs show:

  • No record about an unchanged file was closed (88,724 checks). One revision change closed 5.7% of the current code evidence on average.
  • All 645 supersede edges link to a recorded code change and an earlier trace bound to the same code. The replaced evidence was closed in 624 of 624 cases.

▶ Q3 (W3). Contribution of validation, and direct evidence

Thank you for this insightful comment. We added a new ablation: Full − Validation keeps the Trace Store, both graphs, retrieval, and milestone verification, and turns off only the validity check at recall time (Codex CLI).

BenchmarkBackboneTrace Store + flat retrievalFull − ValidationFull MemTraceGain from validation
DeepSWEDS436.9 ± 2.838.9 ± 2.356.0 ± 1.0+17.1
DeepSWEQwen45.7 ± 1.049.3 ± 1.458.1 ± 1.9+8.8
SWE-EVODS431.9 ± 2.533.3 ± 3.342.7 ± 3.3+9.4
SWE-EVOQwen26.1 ± 3.828.3 ± 2.240.6 ± 3.3+12.3
SWE-MilestoneDS435.6 ± 2.741.3 ± 6.246.4 ± 2.3+5.1
SWE-MilestoneQwen34.3 ± 2.339.5 ± 0.944.3 ± 2.3+4.8

With storage, both graphs, and retrieval unchanged, removing validation lowers the result in all six settings, by 4.8–17.1 points. Storage alone (Trace Store Only in Table 3) is 1.5 points above Native on DeepSWE and SWE-EVO.

Direct evidence:

  • Without validation, closed evidence reaches the agent. In the logs of 268 Full − Validation runs (260 complete), 12 of the 63 recalls about the current workspace state (19.0%) returned closed evidence whose bound file had in fact changed.
  • With validation, it does not directly reach the agent. In Full MemTrace, all 138 recalls of closed evidence carried the historical label.
  • Closed evidence can be wrong. In a Full MemTrace run on SWE-EVO iterative__dvc_1.10.2_1.11.0, eight tests passed, the agent edited dvc/repo/experiments/__init__.py, and MemTrace closed the test result. Replaying the unchanged tests gave 8/8 passes before the edit, 0/8 after it, and 8/8 after restoring only that file. Reusing the earlier result would have certified failing code.
  • Closed evidence is verified again. Of 52 closed passing tests that were rerun, 50 passed again (96.2%) and became current evidence.

These results will be added to the paper.

Thank you again for your time and constructive feedback.

Thank you for the responses to my questions. The new validation ablation and implementation clarifications address several of my concerns and strengthen the empirical evidence. However, the proposed approach still appears largely to integrate existing harness-level memory management and revision-based validation techniques, and its distinctive methodological contribution remains insufficiently established. Therefore, I maintain my rating of 2: Weak Rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.