- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a coding agent can build up knowledge of a large Web application across its releases and use it on later ones. CodeEvoGym turns the release histories of six open-source applications (PHP, TypeScript, Python) into 86 ordered tasks, 60 for training and 26 held out by time. Each task gives the agent the pre-release source and the release's development record (notes, pull requests, issues, discussions) in an air-gapped container; the patch is graded by the tests the release added or modified. During training the agent may query a private oracle that returns pass counts, failing test names and an LLM-written summary of error patterns. Using the Hermes harness, the agent writes three kinds of persistent artifact: memory, skills, and a new documentation store (architecture, journal, lessons). These are frozen and carried into the held-out releases. Across four models, the number of resolved held-out releases does not change with the artifacts (at most 5 of 26). The paper reports a 20–46% relative increase in micro fail-to-pass rate, a 15–25% cost reduction, and journal entries judged faithful on 85 of 86 tasks.
Critical strengths
-
A clean protocol for a question that matters. Each release starts from the true pre-release state, so what carries forward is knowledge and not the agent's earlier code, which separates learning from error propagation. Artifacts are frozen before a temporally later evaluation set, the sandbox is air-gapped, the stateless baseline uses the same harness and budget, and giving one model another model's artifacts (Luna with Sol's store) is an informative design. This is a sound template for studying persistent agent knowledge on evolving code.
-
The unflattering results are reported openly. The paper shows that resolved counts are identical with and without artifacts for all four models, that macro F2P is flat, that two applications get worse, and that one Plane release shapes the aggregate; its conclusion calls the effect "real but modest". Given earlier negative findings on context files and self-written skills, a carefully reported result of this kind has value.
Critical weaknesses
-
The headline effect is, for the one model with a breakdown, produced by a single application, and elsewhere it is zero or negative.
-
The per-application percentages in Figure 3 determine the underlying test counts for GPT 5.6 Sol, and they reproduce the overall figures in Table 1 exactly (65/530 = 12.26%, 95/530 = 17.92%):
Application F2P tests Passing, stateless Passing, with artifacts Change Plane 156 20 53 +33 SuiteCRM 13 4 5 +1 Frappe HRMS 18 5 6 +1 ERPNext 25 16 15 −1 Rocket.Chat 61 15 11 −4 InvoiceNinja 257 5 5 0 Plane contributes +33 and the other five applications −3 in total. One Plane release alone holds 136 of Plane's 156 F2P tests. Without Plane the rate goes from 45/374 (12.0%) to 42/374 (11.2%).
-
Macro F2P, which weights releases equally, is flat or lower for three of four models (35.6 → 35.3, 20.0 → 19.0, 30.4 → 30.0), and the resolved count is unchanged for all four. Results are single runs on 26 tasks with no variance estimate, and two applications hold 413 of the 530 tests that drive the micro rate.
-
The cost claim of "15–25% across models" holds for three of the five artifact conditions: Sol −16.9%, Opus −14.9%, Sonnet −13.4%, Luna with its own artifacts +4.5%, Luna with Sol's artifacts −25.0%. The cost of the 60 training releases is not included.
This is consequential because the supported conclusion is that no effect on correctness is measurable at this sample size, which the conclusion section nearly says but the abstract and contribution list do not. A per-task table, repeated runs, and reporting macro F2P (or capping per-task weight) as the primary metric would make the claims match the evidence.
-
-
The design does not test the paper's own hypothesis, and task validity is not examined.
- The introduction hypothesizes that existing context files fail because they are produced by reading code and never checked against execution. There is no condition with artifacts built without the oracle, and no ablation of the three stores, although the documentation store is listed as a contribution. For RQ1 (accumulation) there is no evidence from the training phase, such as F2P or oracle queries against release index, or a measure of how often stored knowledge goes stale.
- Held-out tasks change 26,040 lines on average and are graded by the tests the release added or modified, with release notes and discussion threads as the specification and 150 model calls as the budget. Tests added in a release typically exercise new internal functions and classes whose names may not appear in that material. InvoiceNinja stays at 1.95% in both conditions (about 5 of 257 tests). The paper does not report what fraction of F2P tests can be satisfied from the description at all. Conversely, the description includes pull-request threads, and whether these contain diffs or code is not stated.
- RQ3 rests on one LLM judge applied to one model's run; usefulness to humans, which the question asks about, is not evaluated.
- Without a task list, release versions or dates, exposure of the held-out releases to the models' training data cannot be assessed, and the environment is described as released without a link.
This is consequential because it limits what the benchmark can show even with more runs: a low ceiling from unsatisfiable tests and the absence of a no-feedback control would both hide the effect the paper is looking for.
Questions for authors
- Can per-task F2P results be reported for all four models, together with the aggregate excluding the 136-test Plane release and with repeated runs? (See Weakness 1.)
- How do artifacts built without the feedback oracle (from reading the code and development record only) compare, and what does each of the three stores contribute separately? (See Weakness 2, first point.)
- What fraction of held-out F2P tests reference identifiers that do not appear in the task description, and is that the reason InvoiceNinja stays near zero? (See Weakness 2, second point.)
- Clarification: does the task description include pull-request diffs or review comments containing code? (See Weakness 2, second point.)
- What does the training phase look like across release order (F2P rate, oracle queries, artifact edits), and what are the versions and dates of the held-out releases? (See Weakness 2, first and last points.)
Author Response - 3.1
We thank the reviewer for the detailed review. We also appreciate the recognition that the protocol separates learning from error propagation, and that the submission reported its unflattering results openly.
The reviewer's reconstruction of the submission's per-application counts is correct:
We agree that the abstract and contributions claimed more than the evidence supported. The general response summarizes our changes and presents the shared results (Tables G1–G13); here we address each point.
W1. Concentration of the effect
On the expanded set, the result no longer rests on micro F2P. With documentation, the three strongest models resolve more held-out releases and improve macro F2P (Tables G3, G4). We now treat macro F2P and resolution as the primary metrics, as the reviewer suggests.
By application. Docs raise macro F2P on most applications and leave P2P unchanged or better everywhere except Plane (Table G5). The gain is positive whichever application is removed (Table G6).
Dominant releases. Table R3.1 removes the releases that dominate micro F2P, including the 136-test Plane release the reviewer identified. The gain remains on both metrics in every case.
Table R3.1. Sensitivity to dominant releases, Opus 5.5, stateless → S+M+D. Δ is computed before rounding.
Per-release results. The gain comes mostly from large releases (Tables G7, G8). Macro F2P is compressed from both ends:
Cost. We withdraw the cost claim. Every self-evolving condition now costs more at evaluation time, and Table G13 adds the training cost, which the reviewer rightly noted was missing. This additional cost mainly comes from the tasks with high f2p tests, where the S+M+D agent performs much better than stateless. We have included a lot more tasks of such categories in the new version of the gym.
W2. Claims and evidence
We agree with the reviewer's reading. The revised abstract and contributions will state the results at the level the evidence supports (G7).
W3. Testing the hypothesis, RQ1, and task validity
What each store contributes. The S+M condition isolates the documentation store (Tables G3, G4). Memory is capped in size and makes up only a few percent of the store, so the contrast is effectively skills vs. skills+docs. On Cal.diy, the application with the longest training history, S+M introduces regressions and roughly doubles cost, while S+M+D does not (Table G9).
RQ1: evidence from the training phase. With the oracle, Opus resolves about half of the training releases, and the lessons and skills record what the oracle reported (Table G12).
Whether F2P tests are satisfiable. About half of the held-out F2P tests go beyond unit level, and none drive a browser (Table G2).
The F2P rate for invoiceninja drops significantly from train to eval. This is because it's release notes and issue/PR discussions directly name only a few functions/classes that are being tested.
Overall, 290 out of 1922 f2p tests (15.1%) in eval are not mentioned in both release notes and issue/PR documents.
What the description contains. The description consists of:
RQ3. We agree that one judge on one model's journals does not establish usefulness to humans, and we have removed that claim. Usefulness to future agents is now measured directly by the store ablation (Table G3).
Task cutoff dates and environment. We have included the cutoff dates for the train/eval split for each repo in our work in Table G1
Answers to questions
We hope these additions address the reviewer's concerns.