- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
MemTrace addresses the problem of recovering useful execution history when a coding agent’s context is refreshed and the repository has changed. It combines immutable execution traces, a Memory Trace Graph, and a Repository State Graph to retrieve and validate historical evidence. Experiments on three benchmarks show improvements in the primary task metrics across two harnesses using one DeepSeek backbone.
Critical strengths
S1. The paper addresses a relevant practical problem. Recovering history is insufficient if that history no longer describes the repository, and explicitly checking evidence before reuse is a sensible direction.
S2. The reported gains are promising, particularly on DeepSWE. Evaluating two harnesses and ablating both graphs provides useful evidence that the system contributes beyond simply storing more history.
Critical weaknesses
W1. The contribution relative to closely related approaches remains unclear. Ledger and MAGE are discussed but not evaluated, while EA-Graph and ContextSniper deserve consideration given their treatment of evidence freshness and repository-synchronized memory. Comparisons against Native, Observation Masking, and RepoGraph do not sufficiently establish what the proposed combination adds over these closer approaches.
W2. The evaluation does not fully establish the claimed mechanism or its generality. All experiments use deepseek-v4-flash-0731-high. The graph ablations are useful, but they do not isolate freshness validation, which is central to the paper’s motivation. The diagnostic counts measure file changes and policy-triggered invalidations rather than independently verified stale-evidence errors. Furthermore, the main performance results lack uncertainty estimates or reported repeated-run variability, and the runtime trade-offs leave the benefit under matched resource budgets unclear.
Questions for authors
Q1. What is the specific methodological distinction from Ledger and EA-Graph, and how does MemTrace compare with a simpler system that combines recoverable history with revision-based invalidation?
Q2. Can you isolate the contribution of validity checking while keeping storage and retrieval unchanged? How often does it prevent reuse of evidence that is actually incorrect or inapplicable?
Q3. Do you have results with another capable backbone? Two harnesses provide useful coverage, but do not establish that the findings generalize across models.
Q4. How many independent runs were performed per configuration, and can you provide uncertainty estimates for the main results and ablations?
The abstract should also qualify “consistently outperforms” as referring to the primary benchmark metrics. Table 2 contains lower Fix and Apply scores, which the main text appropriately acknowledges.
4 comments
▶ Q2 (W2). Isolating validation; how often it prevents reuse of inapplicable evidence
Thank you for this important question.
Ablation (Codex CLI). Full − Validation keeps storage, both graphs, and retrieval unchanged and turns off only validation. "− X" is Full MemTrace without X; "Store + flat" is Trace Store Only in Table 3.
| Benchmark | Backbone | Store + flat | − RSG | − MTG | − Validation | Full MemTrace | Gain |
|---|---|---|---|---|---|---|---|
| DeepSWE | DS4 | 36.9 ± 2.8 | 36.6 ± 0.5 | 39.8 ± 2.3 | 38.9 ± 2.3 | 56.0 ± 1.0 | +17.1 |
| DeepSWE | Qwen | 45.7 ± 1.0 | 47.2 ± 2.6 | 48.1 ± 3.1 | 49.3 ± 1.4 | 58.1 ± 1.9 | +8.8 |
| SWE-EVO | DS4 | 31.9 ± 2.5 | 37.0 ± 3.8 | 34.1 ± 3.3 | 33.3 ± 3.3 | 42.7 ± 3.3 | +9.4 |
| SWE-EVO | Qwen | 26.1 ± 3.8 | 31.2 ± 5.0 | 29.0 ± 2.5 | 28.3 ± 2.2 | 40.6 ± 3.3 | +12.3 |
| SWE-Milestone | DS4 | 35.6 ± 2.7 | 39.1 ± 2.0 | 40.2 ± 5.4 | 41.3 ± 6.2 | 46.4 ± 2.3 | +5.1 |
| SWE-Milestone | Qwen | 34.3 ± 2.3 | 38.4 ± 1.6 | 38.9 ± 4.9 | 39.5 ± 0.9 | 44.3 ± 2.3 | +4.8 |
Removing validation lowers the result in all six settings, by 4.8–17.1 points.
How often. In the logs of 268 Full − Validation runs, 63 recalls asked about the current workspace state, and 12 of them (19.0%) returned closed evidence whose file had in fact changed (10 of 26, 38.5%, among scored runs that were not later rerun). With validation on, all 138 recalls of closed evidence were labeled as history, so none was reused as current.
A verified case. In a run on SWE-EVO iterative__dvc_1.10.2_1.11.0, eight tests passed and the agent then edited dvc/repo/experiments/__init__.py. Replaying the same tests gave 8/8 passes before the edit, 0/8 after it, and 8/8 after restoring only that file, so reusing the earlier result would have certified failing code. Closed evidence is not always wrong: 50 of 52 closed passing tests passed again when rerun.
▶ Q3 (W2). A second backbone
Thank you for this suggestion; we added Qwen3.8-Flash as a second backbone. With Qwen (Qwen columns of the table in Q1), Full MemTrace is 10.9–17.2 points above Native in all six settings, similar to DS4 (9.4–20.6).
▶ Q4 (W2). Number of runs, uncertainty, and resource use
Thank you for raising this point.
- Runs. Every configuration has three runs (one in the submission); all tables report the mean ± standard deviation.
- Consistency. Full MemTrace is above Native in all 12 benchmark–backbone–harness settings, with a mean gain of 14.6 points (95% confidence interval 12.6–16.5).
- Resources. Full MemTrace uses 1.2–1.7 times the tokens of Native. Tokens alone do not explain the gains: Full − Validation uses 0.81–1.15 times the tokens of Full MemTrace and scores 4.8–17.1 points lower.
▶ Abstract. Qualifying "consistently outperforms"
Thank you for this suggestion; we agree. The revised abstract will state that MemTrace outperforms the evaluated baselines on the primary metric of each benchmark (DeepSWE pass@1, SWE-EVO Resolved Rate, SWE-Milestone Score), not on every secondary metric such as SWE-EVO Fix and Apply. The abstract and Tables 2–3 will be updated with the three-run results, and the method section will define each term at first use.
Thank you again for your time and constructive feedback.
Thank you for the additional experiments. The validation ablation, second backbone, and repeated runs meaningfully address several of my concerns and strengthen the empirical evidence.
However, substantial issues remain with the paper’s clarity and presentation. Figures 1 and 2 are overcrowded, with small labels, numerous icons, and competing arrows that make the core mechanism difficult to follow. These figures should help explain the method, but currently add to the reader’s effort. The method description also needs to accurately reflect the implemented validity checks.
Beyond presentation, the distinctive contribution over simpler memory systems with file-level invalidation remains insufficiently established. While the additional results are valuable, I do not think they resolve these concerns sufficiently for acceptance. I am therefore maintaining my rating of 2: Weak rejection
Dear Reviewer 8S2z,
Thank you for the follow-up comment. We agree that the original figures made the mechanism harder to follow. We have uploaded a revised manuscript through the OpenReview Revision option.
The revision makes three targeted changes.
-
Figure 1 (Introduction, p. 2) follows one execution history through refresh and compares Native, Simple Memory, and MemTrace.
-
Figure 2 (Section 3, p. 4) presents three stages: sealing traces, MTG-RSG candidate localization, and the applicability check before reuse.
-
Section 3.3 now matches the implementation. Code observations and changes remain current when their bound entities are unaffected. Tool, test, test-failure, verifier, and requirement-review records close after every workspace revision. There is no per-test dependency tracking. Closed records remain historical, and a closed test result requires fresh verification before current use.
We also want to clarify the remaining distinction from a simpler memory system. The Simple Memory baseline is an append-only execution log with BM25 retrieval and a global workspace-revision filter. It has no execution frontier, milestone relations, MTG-RSG traversal, or type-specific applicability decision. Its global filter is stricter than file-level invalidation and discards unchanged-code records, but does not explain which later trace superseded them. A file-level filter still decides eligibility only, without frontier reconstruction or relation-aware retrieval. MemTrace reconstructs the unfinished frontier, connects repairs and tests through MTG relations, localizes current entities through the RSG, and restores candidates only after type-specific checks. The contribution is this recovery-and-validation mechanism, rather than invalidation alone.
The results support this separation. Under Codex CLI with DS4, Full MemTrace improves over Simple Memory by 19.1 points on DeepSWE (56.0 vs. 36.9), 10.8 on SWE-EVO (42.7 vs. 31.9), and 19.3 on SWE-Milestone (46.4 vs. 27.1). Across six Codex CLI settings, the gain is 10.8–19.3 points. Removing only validation, with storage, both graphs, retrieval, tasks, and budget fixed, lowers scores by 4.8–17.1 points.
The logs provide a direct diagnostic. Without validation, 12/63 recalls about the current workspace (19.0%) returned closed evidence whose bound file had changed. With validation, all 138 closed-evidence recalls carried a historical label. In one SWE-EVO run, eight tests passed, then the agent edited dvc/repo/experiments/__init__.py; replaying them gave 0/8 passes after the edit and 8/8 after restoring only that file. Reusing the earlier result would have certified failing code.
We understand the remaining concern about how much of the gain is distinctive to state-consistent recovery. The revised figures and implementation description make the boundary explicit, while the preceding ablation and Simple Memory results provide quantitative evidence. Although this does not exhaust every file-level baseline, it isolates the added recovery mechanism over the implemented simple baseline. We thank you again for helping us improve the paper.
Dear Reviewer 8S2z,
Thank you for your thoughtful and constructive review. We sincerely appreciate your recognition of the relevance of the problem and of the reported gains, and we take your concerns about closer baselines, the validation mechanism, and generality seriously. We hope the responses below address them, and we welcome any further questions or suggestions.
Terms. A trace is one stored unit of completed work. The Memory Trace Graph (MTG) links traces; the Repository State Graph (RSG) describes current files, symbols, and tests. Validation checks that the code a trace describes is unchanged before reuse. A trace that fails is closed: kept as history, not counted as current evidence.
Setup. Each configuration was run three times; tables report the mean ± standard deviation. Backbones: DS4 (deepseek-v4-flash-0731) and Qwen (Qwen3.8-Flash). Harnesses: Codex CLI and mini-swe-agent. All methods use a 200K-token context window and a 12-hour limit per task. Metrics: DeepSWE pass@1 (%), SWE-EVO Resolved Rate (%), and SWE-Milestone Score. SWE-EVO scores are lower than in the submission (15–17 points for Native): agents sometimes read the benchmark's test files during a run, and the new runs prevent this.
▶ Q1 (W1). Distinction from Ledger and EA-Graph; comparison with a simpler system
Thank you for this helpful question.
All methods. Simple Memory is the simpler system in the question: an append-only history, keyword retrieval, and a rule that accepts only records from the current workspace revision. MAGE and ContextSniper were adapted to coding tasks.
† Python subset (RepoGraph supports Python only): 34 DeepSWE tasks, one SWE-Milestone itinerary.
Full MemTrace is 8.0–19.1 points above the best baseline evaluated on the full task set, in all 12 settings. Simple Memory rejects every earlier record once any file is edited and has no link from a repair to the test that verified it. MAGE's rollback was rarely triggered. Ledger and EA-Graph were not run. These results will be added to the paper.