acceptodds
Under review as a conference paper at ICLR 2027

CodeEvoGym: Learning from Experience Across Software Releases

Abstract

Production web applications never stand still. Each release introduces features, fixes defects, and changes internal structure, making an agent's knowledge of the codebase increasingly outdated. This paper asks whether coding agents can continually update their understanding of a large web application as it evolves, and whether doing so improves their ability to implement changes in unseen releases. We introduce CodeEvoGym, a benchmark and training environment constructed from the release histories of six production open-source web applications spanning three programming languages. Each task provides the source tree immediately preceding a real release, along with its associated development record: release notes, pull requests, issues, and their discussions. The agent's patch is evaluated using the tests added or modified in the post-release source tree. During training, the agent may query a grading oracle that executes the gold test suite in a private container and returns the number and identities of failing tests, together with an LLM-generated summary of their error patterns. The agent distills the resulting feedback into three persistent artifacts, a memory store containing application-specific knowledge, a skill library containing reusable procedures and documentation that is readable by both humans and LLM agents. These artifacts are frozen after the training releases and evaluated on subsequent unseen releases. We examine the impact of using these artifacts on agent performance and cost.

Primary area:
datasets and benchmarks
Keywords:
benchmark, agents, swe agents, self-evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SPLeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a coding agent can build up knowledge of a large Web application across its releases and use it on later ones. CodeEvoGym turns the release histories of six open-source applications (PHP, TypeScript, Python) into 86 ordered tasks, 60 for training and 26 held out by time. Each task gives the agent the pre-release source and the release's development record (notes, pull requests, issues, discussions) in an air-gapped container; the patch is graded by the tests the release added or modified. During training the agent may query a private oracle that returns pass counts, failing test names and an LLM-written summary of error patterns. Using the Hermes harness, the agent writes three kinds of persistent artifact: memory, skills, and a new documentation store (architecture, journal, lessons). These are frozen and carried into the held-out releases. Across four models, the number of resolved held-out releases does not change with the artifacts (at most 5 of 26). The paper reports a 20–46% relative increase in micro fail-to-pass rate, a 15–25% cost reduction, and journal entries judged faithful on 85 of 86 tasks.

Critical strengths
  1. A clean protocol for a question that matters. Each release starts from the true pre-release state, so what carries forward is knowledge and not the agent's earlier code, which separates learning from error propagation. Artifacts are frozen before a temporally later evaluation set, the sandbox is air-gapped, the stateless baseline uses the same harness and budget, and giving one model another model's artifacts (Luna with Sol's store) is an informative design. This is a sound template for studying persistent agent knowledge on evolving code.

  2. The unflattering results are reported openly. The paper shows that resolved counts are identical with and without artifacts for all four models, that macro F2P is flat, that two applications get worse, and that one Plane release shapes the aggregate; its conclusion calls the effect "real but modest". Given earlier negative findings on context files and self-written skills, a carefully reported result of this kind has value.

Critical weaknesses
  1. The headline effect is, for the one model with a breakdown, produced by a single application, and elsewhere it is zero or negative.

    • The per-application percentages in Figure 3 determine the underlying test counts for GPT 5.6 Sol, and they reproduce the overall figures in Table 1 exactly (65/530 = 12.26%, 95/530 = 17.92%):

      ApplicationF2P testsPassing, statelessPassing, with artifactsChange
      Plane1562053+33
      SuiteCRM1345+1
      Frappe HRMS1856+1
      ERPNext251615−1
      Rocket.Chat611511−4
      InvoiceNinja257550

      Plane contributes +33 and the other five applications −3 in total. One Plane release alone holds 136 of Plane's 156 F2P tests. Without Plane the rate goes from 45/374 (12.0%) to 42/374 (11.2%).

    • Macro F2P, which weights releases equally, is flat or lower for three of four models (35.6 → 35.3, 20.0 → 19.0, 30.4 → 30.0), and the resolved count is unchanged for all four. Results are single runs on 26 tasks with no variance estimate, and two applications hold 413 of the 530 tests that drive the micro rate.

    • The cost claim of "15–25% across models" holds for three of the five artifact conditions: Sol −16.9%, Opus −14.9%, Sonnet −13.4%, Luna with its own artifacts +4.5%, Luna with Sol's artifacts −25.0%. The cost of the 60 training releases is not included.

    This is consequential because the supported conclusion is that no effect on correctness is measurable at this sample size, which the conclusion section nearly says but the abstract and contribution list do not. A per-task table, repeated runs, and reporting macro F2P (or capping per-task weight) as the primary metric would make the claims match the evidence.

  2. The design does not test the paper's own hypothesis, and task validity is not examined.

    • The introduction hypothesizes that existing context files fail because they are produced by reading code and never checked against execution. There is no condition with artifacts built without the oracle, and no ablation of the three stores, although the documentation store is listed as a contribution. For RQ1 (accumulation) there is no evidence from the training phase, such as F2P or oracle queries against release index, or a measure of how often stored knowledge goes stale.
    • Held-out tasks change 26,040 lines on average and are graded by the tests the release added or modified, with release notes and discussion threads as the specification and 150 model calls as the budget. Tests added in a release typically exercise new internal functions and classes whose names may not appear in that material. InvoiceNinja stays at 1.95% in both conditions (about 5 of 257 tests). The paper does not report what fraction of F2P tests can be satisfied from the description at all. Conversely, the description includes pull-request threads, and whether these contain diffs or code is not stated.
    • RQ3 rests on one LLM judge applied to one model's run; usefulness to humans, which the question asks about, is not evaluated.
    • Without a task list, release versions or dates, exposure of the held-out releases to the models' training data cannot be assessed, and the environment is described as released without a link.

    This is consequential because it limits what the benchmark can show even with more runs: a low ceiling from unsatisfiable tests and the absence of a no-feedback control would both hide the effect the paper is looking for.

Questions for authors
  1. Can per-task F2P results be reported for all four models, together with the aggregate excluding the 136-test Plane release and with repeated runs? (See Weakness 1.)
  2. How do artifacts built without the feedback oracle (from reading the code and development record only) compare, and what does each of the three stores contribute separately? (See Weakness 2, first point.)
  3. What fraction of held-out F2P tests reference identifiers that do not appear in the task description, and is that the reason InvoiceNinja stays near zero? (See Weakness 2, second point.)
  4. Clarification: does the task description include pull-request diffs or review comments containing code? (See Weakness 2, second point.)
  5. What does the training phase look like across release order (F2P rate, oracle queries, artifact edits), and what are the versions and dates of the held-out releases? (See Weakness 2, first and last points.)
1 comment
AuthorsOpenReview ↗

Author Response - 3.1

We thank the reviewer for the detailed review. We also appreciate the recognition that the protocol separates learning from error propagation, and that the submission reported its unflattering results openly.

The reviewer's reconstruction of the submission's per-application counts is correct:

  • the micro F2P gain came from Plane;
  • macro F2P was flat;
  • resolution did not change;
  • the cost claim held for only some conditions.

We agree that the abstract and contributions claimed more than the evidence supported. The general response summarizes our changes and presents the shared results (Tables G1–G13); here we address each point.

W1. Concentration of the effect

On the expanded set, the result no longer rests on micro F2P. With documentation, the three strongest models resolve more held-out releases and improve macro F2P (Tables G3, G4). We now treat macro F2P and resolution as the primary metrics, as the reviewer suggests.

By application. Docs raise macro F2P on most applications and leave P2P unchanged or better everywhere except Plane (Table G5). The gain is positive whichever application is removed (Table G6).

Dominant releases. Table R3.1 removes the releases that dominate micro F2P, including the 136-test Plane release the reviewer identified. The gain remains on both metrics in every case.

Table R3.1. Sensitivity to dominant releases, Opus 5.5, stateless → S+M+D. Δ is computed before rounding.

ExcludedReleases leftF2P macroΔF2P microΔ
none4335.0 → 42.3+7.316.8 → 28.4+11.7
plane v1.4.0-rc1 (136 F2P tests)4235.8 → 41.6+5.818.0 → 25.4+7.3
All of Plane4135.4 → 40.2+4.817.7 → 24.6+6.9
Plane + Cal.diy2936.0 → 39.1+3.110.0 → 13.7+3.6
10 largest releases3339.4 → 44.4+5.029.0 → 36.7+7.7

Per-release results. The gain comes mostly from large releases (Tables G7, G8). Macro F2P is compressed from both ends:

  • two of the largest releases stay unsolved by every condition (Table G8);
  • most of the smallest releases are already fully solved without self-evolution (Table G7).

Cost. We withdraw the cost claim. Every self-evolving condition now costs more at evaluation time, and Table G13 adds the training cost, which the reviewer rightly noted was missing. This additional cost mainly comes from the tasks with high f2p tests, where the S+M+D agent performs much better than stateless. We have included a lot more tasks of such categories in the new version of the gym.

W2. Claims and evidence

We agree with the reviewer's reading. The revised abstract and contributions will state the results at the level the evidence supports (G7).

W3. Testing the hypothesis, RQ1, and task validity

What each store contributes. The S+M condition isolates the documentation store (Tables G3, G4). Memory is capped in size and makes up only a few percent of the store, so the contrast is effectively skills vs. skills+docs. On Cal.diy, the application with the longest training history, S+M introduces regressions and roughly doubles cost, while S+M+D does not (Table G9).

RQ1: evidence from the training phase. With the oracle, Opus resolves about half of the training releases, and the lessons and skills record what the oracle reported (Table G12).

Whether F2P tests are satisfiable. About half of the held-out F2P tests go beyond unit level, and none drive a browser (Table G2).

The F2P rate for invoiceninja drops significantly from train to eval. This is because it's release notes and issue/PR discussions directly name only a few functions/classes that are being tested.

Overall, 290 out of 1922 f2p tests (15.1%) in eval are not mentioned in both release notes and issue/PR documents.

What the description contains. The description consists of:

  • the release notes;
  • the pull requests and the issues they close;
  • the discussion on both, including inline review comments. It does not include diffs.

RQ3. We agree that one judge on one model's journals does not establish usefulness to humans, and we have removed that claim. Usefulness to future agents is now measured directly by the store ablation (Table G3).

Task cutoff dates and environment. We have included the cutoff dates for the train/eval split for each repo in our work in Table G1

Answers to questions

  1. Per-release results and the aggregate without the 136-test Plane release. See Tables G5–G8, R3.1.
  2. Per-store contributions. The S+M vs. S+M+D contrast is in Tables G3, G4 and G9.
  3. Identifiers missing from descriptions, and InvoiceNinja. InvoiceNinja is explained by a single release (Table G8).
  4. Pull-request diffs. Diffs are not included; review comments are (see W3).
  5. Training dynamics, versions and dates. Training results are in Table G12, while cutoff dates are in Table G1.

We hope these additions address the reviewer's concerns.

Reviewer_e3uEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to assess the capabilities of frontier models in continually understanding and developing large web applications that evolve over time. To achieve this, the authors propose a benchmark involving six production open-source web applications, whose official releases are partitioned along timeline into training and test splits. During training, the agent is allowed to update its skill, memory and document, which are later frozen and carried into the testing. Frontier models including GPT and Claude are evaluated on the proposed CodeEvoGym benchmark and their performances are all low in terms of resolution rates.

Critical strengths

It is a reasonable idea to use releases to build large-scale coding tasks by collecting the corresponding PRs and issues and simulate the codebase evolution with continual updates.

The agent evolution is a significant research topic and the proposed benchmark provides an option for the corresponding evaluation.

Critical weaknesses

The value of the benchmark is limited.

  1. The scope of task is quite narrow, only involving web applications. The number of tasks is small.
  2. The grading of tasks mainly depends on UTs. But web applications should also involve UI operations. It is unclear how the authors could ensure a comprehensive and reliable verification of task completion.
  3. It is not verified that the tasks are solvable with the provided PRs and issues. Common issues include inconsistency between UTs and task instructions, unknowing exact interfaces to be tested, etc.

Evaluation of agent evolution is weak.

  1. Only use Hermes Agent. Other typical harnesses or methods like DSH are not evaluated.
  2. No repeated runs to enhance the stability of results.

Analysis of experimental results are weak.

  1. The text mainly "repeats" the on-surface phenomenon with little in-depth analysis and insights.
  2. Little, if not none, study of concrete cases, failures, success, etc.
Questions for authors

Some limitations have been identified by the authors themselves. The reviewer hopes the authors could resolve some important ones before submitting the manuscript.

2 comments
AuthorsOpenReview ↗

Author Response - 2.1

We thank the reviewer for the thoughtful review. We appreciate the recognition that building release-level tasks from releases, pull requests and issues is a reasonable way to simulate codebase evolution, and that agent self-evolution is an important topic.

The reviewer raised three groups of concerns:

  • the value of the benchmark: scope, grading by unit tests, and solvability;
  • the evaluation of agent evolution: a single harness and single runs;
  • the depth of the analysis.

The general response summarizes our changes and presents the shared results (Tables G1–G13); here we address each point.

W1. Scope and size of the benchmark

Our focus on web applications is deliberate. Existing benchmarks are dominated by libraries, while production web applications are long-lived systems whose behaviour spans HTTP interfaces, databases and background services. The nine applications cover invoicing, CRM, accounting, asset management, chat, project management, scheduling, ERP and HR, across three languages.

Each task is a full release rather than a single issue, so each one bundles many changes and many tests (Table G1). We agree the number of applications is still modest and state this as a limitation.

W2. Grading beyond unit tests

The reviewer is right that web applications also have UI behaviour. Table G2 shows what the held-out F2P tests actually run. About half go beyond unit level:

  • Laravel feature tests send HTTP requests through the application kernel, including Snipe-IT's UI routes.
  • API contract tests exercise the HTTP surface.
  • Frappe tests run against a live site database.
  • Cal.diy's component tests render React components.

None of the F2P tests drive a browser. We now state explicitly that CodeEvoGym grades server-side and component-level behaviour, not end-to-end UI interaction.

W3. Solvability of tasks

Every F2P test fails on the pre-release code and passes on the real post-release code, so each one can be reached by code changes alone. The reviewer's concern is a different one: whether the description gives enough information to reach it, for example when tests call interfaces the release notes never name. We agree this is not guaranteed, and partial-credit scoring is designed for exactly this case.

Two observations bound the problem:

  • With feedback. During training, the oracle names failing tests and their error messages, and agents resolve a large share of training releases (Table G12). Most mismatches can therefore be overcome once the interface is surfaced.
  • Without feedback. At evaluation, a few of the largest releases stay unsolved by every condition (Table G8). This is where the problem bites. For example, this problem is most prominent in the outlier repo, invoiceninja. The F2P rate for invoiceninja drops significantly from train to eval. This is because it's release notes and issue/PR discussions directly name only a few functions/classes that are being tested.

Overall, 290 out of 1922 f2p tests (15.1%) in eval are not mentioned in both release notes and issue/PR documents.

E1. A single harness

We use one harness so that conditions differ only in their stores. Within it we compare two self-evolution designs against a stateless baseline: Hermes's own skill and memory stores (S+M), and the same stores plus our documentation store (S+M+D). The two designs behave very differently (Tables G3, G4). The environment itself is harness-agnostic: a container per task, a grading oracle, and file-based stores that any harness can read and write.

A1. Where the gains come from, and why

We agree that the submission mostly described aggregate numbers. The revised analysis explains the results:

  • Release size. The gain comes mostly from large and mid-sized releases, where knowing the codebase matters most; the smallest releases are near ceiling (Tables G7, G8).
  • Behaviour. Both self-evolving agents run their own tests far more often. Only docs improve localization: with docs, edits target the functions the release changed more precisely, while skills alone produce larger patches without better targeting (Tables G10, G11).
  • Regressions. On the application with the longest training history, skills alone introduce regressions that docs avoid (Table G9).
AuthorsOpenReview ↗

Author Response - 2.2

A2. Case studies

Learning from feedback during training. Table R2.1 follows one training release, which we will also include in the Appendix.

  • The first oracle report showed that the agent's change broke many existing tests with the same error. The oracle's summary pointed to state that was not initialized on every code path.
  • The agent fixed the regressions, then fixed an ordering error in the tax calculation, and resolved the release.
  • It recorded the general lesson that optional state must be guarded at every consumer, not only where it is produced.

Table R2.1. Oracle feedback during one training release.

TurnOracle feedbackAgent's next step
47F2P 1/3; P2P 16/34 (18 regressions, same missing-attribute error)Guard the downstream consumer of the new state
55F2P 1/3; P2P 34/34Evaluate non-tax deductions before the annual tax computation
81F2P 3/3; P2P 34/34; resolvedWrite journal, architecture, lessons, skill and memory entries

We hope these additions address the reviewer's concerns, and we would welcome further questions.

Reviewer_Go4YOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CodeEvoGym studies experience transfer across releases of six production web applications, using 60 training and 26 evaluation tasks. Each task starts from the actual pre-release source tree, while agent-written knowledge artifacts carry forward. Training provides private execution feedback; evaluation proceeds without oracle access. Experiments report improvements in test-weighted partial progress, heterogeneous application-level effects, and largely unchanged full-release resolution. An LLM judge additionally evaluates journal faithfulness.

Critical strengths
  1. Starting each task from the true pre-release snapshot while transferring knowledge artifacts usefully separates experience transfer from accumulated implementation errors.
  2. Release histories, applications spanning three languages, containerized execution, a private oracle, and temporal evaluation provide a practically relevant environment for studying reusable repository knowledge.
  3. Reporting separates partial progress, resolution, regressions, and cost. Application-level results and stronger-to-weaker artifact transfer expose meaningful tradeoffs rather than suggesting uniformly beneficial transfer.
Critical weaknesses
  1. The main empirical benefit is insufficiently robust. Evaluation includes only 26 releases, and Plane contributes more newly passing tests than the overall net gain, partly through one release containing 136 F2P tests. Repeated runs and sensitivity to dominant releases would strengthen the conclusions.
  2. Missing component and protocol ablations leave the mechanism unclear. The experiments do not adequately distinguish memory, skills, and documentation, or separate chronological experience from oracle feedback and additional repository exploration.
  3. The benchmark’s distinctive contribution needs sharper substantiation. The related-work discussion identifies protocol differences, but the experiments only partially establish what additional insight specifically requires their integration.
  4. Measurement reliability needs stronger support, including test stability, exclusions, and coverage of release requirements. P2P checks concern selected added or modified tests, limiting conclusions about broader regression preservation. Reported evaluation-cost savings also do not establish amortized savings after artifact construction and oracle use.
  5. The documentation evaluation supports a narrower conclusion than practical usefulness. The 85/86 faithfulness result uses one LLM judge on one model’s journals; it does not establish whether the documentation is useful to humans or future agents.
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Author Response - 1.1

We thank the reviewer for the careful reading. We also appreciate the recognition that starting each task from the true pre-release state, using a temporal protocol, and reporting progress, resolution, regressions and cost separately are useful design choices. The reviewer's main concerns were that the evaluation set is small and dominated by one release, that ablations are missing, and that the cost and documentation claims go beyond the evidence. We think these concerns were justified. The general response summarizes our changes and presents the shared results (Tables G1–G13 in general comment above); here we address each point.

W1. Robustness: small evaluation set, Plane dominance

The held-out set is now larger and spans more applications (Tables G1, G5). We now treat macro F2P, which weights each release equally, as the primary partial-credit metric alongside resolution; micro F2P is secondary. Plane now contributes under a tenth of the held-out F2P tests.

The gain is positive whichever application is removed (Table G6). Plane is the most influential application, but removing Plane and Cal.diy together, which account for most of the newly passing tests, still leaves a positive gain on both metrics. Per release, the gain comes mostly from large and mid-sized releases (Tables G7, G8).

W2. Component and protocol ablations

Docs vs. skills and memory. S+M and S+M+D differ only in whether the docs store exists (Tables G3, G4). Memory is capped in size and repository-agnostic by design, and it makes up only a few percent of the store. The contrast is therefore effectively skills vs. skills+docs.

S+M does raise Opus's micro F2P without raising macro F2P or resolution. Outside Cal.diy, that gain is concentrated in a few test-heavy releases while macro F2P falls. This is the same concentration pattern the reviewer flagged in the submission. On Cal.diy, S+M introduces regressions and roughly doubles cost, while S+M+D improves F2P and keeps P2P intact (Table G9).

Mechanism. Self-evolving agents run their own tests far more often, but only docs improve where edits land (Tables G10, G11). The S+M agent writes much larger patches without editing the functions the release changed any more precisely. With docs, edits target those functions more precisely, so docs act mainly on localization.

Additional exploration. The S+M agent makes a similar number of tool calls and reads as much code as the S+M+D agent, yet does not improve (Table G10). Additional exploration therefore does not explain the gain.

Oracle feedback vs. chronological experience. With the oracle, Opus resolves about half of the training releases (Table G12). The lessons and skills record what the oracle reported. Appendix A traces one release from partial to full resolution through oracle feedback, together with the lessons and skill entries this produced.

W3. What the integrated protocol adds

Existing release-level benchmarks either score each release on its own (SWE-EVO), which leaves transfer unmeasured, or chain the agent's own outputs from one release to the next (SWE-Chain, EvoClaw), which mixes transfer with error propagation. Self-evolution benchmarks such as Evo-Memory and SkillsBench, meanwhile, keep the environment fixed. CodeEvoGym combines what each of these lacks. Every release starts from the true pre-release code, so any gain has to come from what the agent stored rather than code it inherited. The stores are frozen before evaluation on later, larger releases, and the private oracle lets agents ground what they write in actual test outcomes. Together, these choices are what make the new results measurable. They show that documentation helps where skills and memory alone do not (Tables G4, G11), and that a weaker model benefits from a stronger model's store but not from its own (Table G4). They also show that skills accumulated over a long release history can introduce regressions that documentation avoids, as on Cal.diy (Table G9). We will make this point more explicit in the camera-ready version.

W4. Measurement reliability and cost

Test coverage. Table G2 shows what the held-out F2P tests run. About half go beyond unit level, and none are browser end-to-end tests.

P2P scope. We agree that P2P only covers previously passing tests in the files the release added or modified, so it cannot establish broader regression safety. We state this as a limitation.

Cost. We withdraw the cost-saving claim. Every self-evolving condition costs more at evaluation time, and Table G13 adds the one-off training cost.

AuthorsOpenReview ↗

Author Response - 1.2

W5. Documentation evaluation

We agree, and have narrowed RQ3.

Usefulness to future agents is now measured directly by the ablation. Adding the documentation store to S+M, with everything else fixed, raises both resolution and macro F2P for Opus and Sol (Table G3). Agents do read the docs, and reading them coincides with more precise edits (Tables G10, G11).

Usefulness to humans is not established by these results. We have removed that claim from RQ3 and the conclusion.

Thank you for the substantial revisions. The documentation ablation and the narrower cost and usefulness claims are meaningful improvements.

However, I am not yet convinced that the revised evidence establishes sufficiently robust and explanatory findings. More precise edits support the localization interpretation, but do not establish it causally. Similar exploration counts also do not fully isolate the source of improvement, while the oracle examples leave the role of chronological accumulation unclear. The test-coverage breakdown clarifies scope, but does not by itself establish scoring reliability.

For now, I retain my score of 1 (Reject). I will reconsider this assessment after examining the magnitude, distribution, and uncertainty of the gains in the shared tables.

Thank you for your response. We look forward to seeing your final review score, after your detailed analysis of our results. We hope we addressed your concerns. Thanks for your detailed comments in the review. They greatly helped us in designing the new experiments.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for careful and constructive reviews. Several concerns were shared:

  • the evaluation set was small and its aggregate was shaped by a single release;
  • the memory, skill and documentation stores were not ablated;
  • the cost and documentation claims went beyond the evidence;
  • it was unclear what the hidden tests actually verify.

We think these concerns were justified and we have substantially extended the study with this in mind:

  1. Larger benchmark. The benchmark grows from 86 to 140 release tasks (+62%) and from 6 to 9 applications (adding Akaunting, Snipe-IT and Cal.diy, +50%). The held-out set grows from 26 to 43 releases (+65%).
  2. Component ablation. We add a Skills+Memory (S+M) condition, which uses only the stores the Hermes harness provides. Comparing it with Skills+Memory+Docs (S+M+D) isolates the documentation store.
  3. Process-level analyses. We add per-release results, tool-call breakdowns, code analysis against the reference diff, training-phase results, and cost accounting that includes training.
  4. Revised claims. The expanded results change our headline (G2), and we withdraw the cost-reduction claim.

This response presents the results shared across reviews (Tables G1–G13). Each reviewer response refers to them and addresses the remaining points.

G1. Expanded benchmark

Table G1. Composition of the benchmark. F2P and P2P are per-release averages.

ApplicationLang.TrainEvalF2P / task (train)F2P / task (eval)P2P / task (train)P2P / task (eval)Eval Cutoff
InvoiceNinjaPHP10414.558.2105.4110.0Apr 2026
SuiteCRMPHP6213.85.0135.0224.5Apr 2019
AkauntingPHP9423.120.232.423.2May 2023
Snipe-ITPHP10517.5151.053.7141.0May 2026
Rocket.ChatTS10518.313.427.026.6Feb 2026
PlaneTS528.877.535.891.5Jul 2026
Cal.diyTS271227.948.563.5124.0Nov 2025
ERPNextPython1257.25.0139.1209.2Oct 2024
Frappe HRMSPython844.53.5115.655.0Jan 2024
All974317.5544.7076.96110.63

Each task is a full release rather than a single issue. Held-out releases are larger than training releases on average, so the held-out set tests transfer to harder releases.

Table G2. Held-out F2P tests by type.

Test typeWhat runsApplicationsF2P tests
UnitFunctions and classes in-process, no HTTP server (SuiteCRM's hit the DB through SugarBeans)cal.diy, Rocket.Chat, Plane, InvoiceNinja, Snipe-IT, Akaunting, SuiteCRM962
Laravel FeatureHTTP requests through the app kernel, artisan commands and models against a test DB; Snipe-IT splits these into Api/ and Ui/Snipe-IT, Akaunting, InvoiceNinja784
API contractHTTP-level tests of the API surface (DRF test client + DB; codeception against the running app)Plane, SuiteCRM65
IntegrationE-invoicing router; CRM service clientsInvoiceNinja, cal.diy51
Frappe DB integrationDoctype test classes against a live site DBERPNext, HRMS39
React componentComponents rendered in jsdomcal.diy21
Browser e2e (Playwright/Cypress)——0
Total1,922

About half of the held-out F2P tests go beyond unit level. They send HTTP requests through the application kernel, test API contracts, run against live databases, or render React components. None are browser end-to-end tests. We state this as a limitation.

AuthorsOpenReview ↗

General response to all reviewers - 2

G2. Revised main result

Table G3. Results on the 43 held-out releases from 9 applications.

ModelSelf-evolutionResolvedF2P macroF2P microP2P macroP2P microEval cost
Claude Opus 5.5None5/4335.0016.7597.0096.50$81.71
S+M5/4333.0422.8196.9098.00$138.15
S+M+D8/4342.3128.4399.5799.47$115.01
GPT 5.6 SolNone4/4329.9218.5299.3199.16$172.54
S+M3/4326.9017.1799.5599.26$186.98
S+M+D5/4332.5220.8698.4798.28$220.52
Claude Sonnet 5None1/4313.007.4499.8599.87$126.70
S+M+D3/4321.4910.8799.8599.85$154.37
GPT 5.6 LunaNone2/4315.025.9899.1297.96$2.83
S+M+D (own)2/4314.016.1497.7396.62$4.04
S+M+D (Sol's)2/4315.807.1399.2599.45$3.74

Table G4. Change relative to the no-self-evolution agent (points; cost in %).

ModelConditionΔ ResolvedΔ F2P macroΔ F2P microΔ P2P microΔ Cost
Opus 5.5S+M0−2.0+6.1+1.5+69%
Opus 5.5S+M+D+3+7.3+11.7+3.0+41%
SolS+M−1−3.0−1.4+0.1+8%
SolS+M+D+1+2.6+2.3−0.9+28%
Sonnet 5S+M+D+2+8.5+3.40.0+22%
LunaS+M+D (own)0−1.0+0.2−1.3+43%
LunaS+M+D (Sol's)0+0.8+1.2+1.5+32%

Three patterns hold across models (Table G4).

Documentation is the component that makes self-evolution pay off. With S+M+D, the three strongest models resolve more held-out releases and improve macro F2P. P2P improves for Opus and is essentially unchanged for Sonnet and Sol.

Skills and memory alone do not help. For both models where we ran S+M, it never improves resolution or macro F2P, and it always raises cost.

Self-evolution is not free. S+M+D costs more than the stateless agent at evaluation time for every model, before counting the one-off training phase (G6).

The weakest model, Luna, gains nothing from the store it writes itself, but it gains modestly from Sol's store. The value of a store therefore depends on who wrote it.

The S+M result agrees with two recent studies. SkillsBench (Li et al., 2026a) finds that skills an agent writes for itself score below the no-skills baseline, even though curated skills help substantially. Hajimiri et al. (2026) find that on web tasks, a vanilla agent given the same token budget matches or beats online skill and memory modules while often using fewer tokens, and that weaker models write noisier modules. They also note that environments where later tasks depend on knowledge accumulated from earlier ones could favour such modules. A single codebase evolving across releases is such an environment. Our results suggest that what pays off there is a descriptive account of the codebase, not self-written procedures.

AuthorsOpenReview ↗

General response to all reviewers - 3

G3. Robustness of the gain

We now treat macro F2P, which weights each release equally, as the primary partial-credit metric alongside resolution; micro F2P is secondary. Tables G5–G8 use Claude Opus 5.5.

Table G5. Results by application, stateless → S+M+D. Eval F2P counts and Δ passing tests are derived from per-release averages and per-application rates.

ApplicationLang.Train / evalEval F2P testsF2P macroF2P microΔ passing F2PP2P micro
InvoiceNinjaPHP10 / 42330.88 → 1.171.29 → 2.15+2100.00 → 100.00
SuiteCRMPHP6 / 21066.67 → 100.0060.00 → 100.00+4100.00 → 100.00
AkauntingPHP9 / 48128.57 → 16.0716.05 → 11.11−4100.00 → 100.00
Snipe-ITPHP10 / 575512.56 → 20.887.81 → 12.05+3280.43 → 100.00
Rocket.ChatTS10 / 56714.09 → 38.6417.91 → 35.82+12100.00 → 100.00
PlaneTS5 / 215526.32 → 84.196.45 → 72.26+102100.00 → 96.17
Cal.diyTS27 / 1258234.08 → 42.9733.16 → 46.74+7998.12 → 98.79
ERPNextPython12 / 52566.67 → 66.6760.00 → 60.00099.90 → 100.00
Frappe HRMSPython8 / 41481.25 → 58.3378.57 → 57.14−399.55 → 100.00
All97 / 431,92234.98 → 42.2616.75 → 28.41+22496.47 → 99.47

Docs raise macro F2P on most applications. They leave ERPNext unchanged and lower it on Frappe HRMS and Akaunting. P2P is unchanged or better on every application except Plane.

Table G6. Leave-one-application-out, stateless → S+M+D. Δ is computed before rounding.

ExcludedF2P macroΔF2P microΔ
none (all 43)35.0 → 42.3+7.316.8 → 28.4+11.7
InvoiceNinja38.5 → 46.5+8.018.9 → 32.0+13.1
SuiteCRM33.4 → 39.4+6.016.5 → 28.0+11.5
Akaunting35.6 → 44.9+9.316.8 → 29.2+12.4
Snipe-IT37.9 → 45.1+7.122.5 → 39.0+16.5
Rocket.Chat37.7 → 42.7+5.016.7 → 28.1+11.4
Plane35.4 → 40.2+4.817.7 → 24.6+6.9
Cal.diy35.3 → 42.0+6.79.6 → 20.4+10.8
ERPNext30.8 → 39.1+8.216.2 → 28.0+11.8
Frappe HRMS30.2 → 40.6+10.416.3 → 28.2+11.9
Plane + Cal.diy36.0 → 39.1+3.110.0 → 13.7+3.6

The gain is positive in every case. Plane is the most influential application. Removing Plane and Cal.diy together, which account for most of the newly passing tests, still leaves a positive gain on both metrics.

Table G7. Results by release size, stateless → S+M+D. Releases are ranked by number of F2P tests.

ReleasesCountF2P testsF2P macroF2P microΔ passing F2P
10 largest101,36420.3 → 35.211.7 → 25.0+181
Middle2351928.0 → 36.226.6 → 34.7+42
10 smallest103965.8 → 63.361.5 → 64.1+1
All431,92235.0 → 42.316.8 → 28.4+224

Table G8. The 10 largest held-out releases.

ReleaseF2P testsStatelessS+M+DΔ
snipe-it v8.6.04470 (0.0%)0 (0.0%)0
invoiceninja v5.13.221752 (1.1%)4 (2.3%)+2
snipe-it v8.6.216924 (14.2%)39 (23.1%)+15
plane v1.4.0-rc11360 (0.0%)93 (68.4%)+93
cal.diy v6.1.01084 (3.7%)69 (63.9%)+65
cal.diy v6.1.38346 (55.4%)46 (55.4%)0
snipe-it v8.7.27235 (48.6%)35 (48.6%)0
cal.diy v6.1.126246 (74.2%)46 (74.2%)0
akaunting 3.1.0560 (0.0%)0 (0.0%)0
cal.diy v6.0.12563 (5.4%)9 (16.1%)+6
Total1,364160 (11.7%)341 (25.0%)+181

The gain comes mostly from the largest and mid-sized releases, where knowing the codebase matters most. The smallest releases are near ceiling: most are already fully solved without self-evolution, which compresses macro F2P. Two of the largest releases stay unsolved by every condition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.