acceptodds
Under review as a conference paper at ICLR 2027

History-Aware and Scope-Constrained Skill Evolution for Coding Agents

Abstract

Large language model (LLM)-based coding agents have made substantial progress in automated software engineering, and recent studies further enable them to accumulate reusable knowledge from completed task trajectories. However, as code repositories continuously evolve, accumulated knowledge may become outdated or even mislead agents on new tasks, making it necessary for knowledge to evolve alongside the repository. Existing knowledge self-evolution methods typically update knowledge based on its current state and newly acquired experience, overlooking both the history of successive updates and the scope within which knowledge remains applicable. This leads to knowledge degeneration as repeated updates make knowledge increasingly redundant or overly task-specific, and cross-repository contamination as repository-specific experience is inappropriately used to modify knowledge beyond its applicable scope. To address these limitations, we propose EvoGraph, a history-aware and scope-constrained framework for knowledge self-evolution in coding agents that represents reusable knowledge as skills. To mitigate knowledge degeneration, EvoGraph maintains an Evolution History Graph (EHG) that preserves historical skill versions and their revision, derivation, and consolidation relations, enabling proposed updates to be assessed against earlier versions and evolution-related skills. To reduce cross-repository contamination, the EHG further records the repository scope of each skill, based on which EvoGraph constrains both skill retrieval and skill updates, restricting inappropriate knowledge reuse and modification across repositories. Experiments on SWE-bench Verified and Pro with Qwen3.6-27B, GPT-5 mini, and DeepSeek-V4-Flash show that EvoGraph achieves average relative improvements of 6.90%–14.04% in resolution rate over all baselines across the three models. All code and data are available at https://anonymous.4open.science/r/EvoGraph-B1C4.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agent, Skill Evolution, Evolution History Graph

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UuKcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoGraph, a framework for continual knowledge evolution in coding agents. It organizes reusable skills in an Evolution History Graph that records historical versions, evolution relations, and repository scope. Based on this graph, EvoGraph performs scope-constrained retrieval and history-aware maintenance to reduce knowledge degeneration and cross-repository contamination. Experiments on SWE-bench Verified and SWE-bench Pro with three backbone LLMs show consistent improvements over the evaluated baselines.

Critical strengths
  1. The paper identifies knowledge degeneration and cross-repository contamination as two practical issues and introduces corresponding mechanisms.

  2. EvoGraph performs consistently well across two SWE benchmarks and three backbone models.

  3. Historical skill versions, evolution relations, and repository scope are explicitly modeled in a unified framework.

Critical weaknesses
  1. Closely related works have explored persistent decision history, skill versioning, and maintenance under evolving repositories, including SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History (arXiv, 2026), AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolution (arXiv, 2026), and Repo2Skill-Evo: Repository Skills Go Stale in Silence (arXiv, 2026). The methodological distinctions and additional contributions of EvoGraph relative to these works are not sufficiently clarified.

  2. The "w/o History-Aware Maintenance" variant freezes the EHG entirely, removing both historical information and continual adaptation, making it difficult to isolate the contribution of history-aware maintenance from online skill updating.

  3. The reduction in knowledge redundancy may also result from the skip, update, create, and consolidation operations, making it unclear how much of the improvement can be attributed specifically to evolution history.

  4. EvoGraph incurs additional token consumption across several settings, but the paper does not provide budget-matched comparisons or a systematic analysis of the resolution–cost tradeoff.

Questions for authors
  1. Could the authors add a current-state-only variant that continues online skill updating but does not access historical versions?

  2. Could the authors clarify how EvoGraph differs from closely related methods such as CODESKILL, AutoSkill, and SkillHone?

  3. Could the authors further separate the effect of evolution history from skip, update, create, and consolidation when analyzing knowledge degeneration?

  4. Could the authors provide an equal-budget comparison or a resolution-versus-token curve to better evaluate the tradeoff between performance and inference cost?

5 comments
AuthorsOpenReview ↗

Response to the Review [1/2]

We thank the reviewer for the constructive feedback and the opportunity to clarify the following points:

Q1 / W2: Contribution of Evolution History

We construct a current-state-only variant, w/o Evolution History, which continues online skill updating but withholds historical versions and evolution relations from the skill editor, so that each update is based only on the current state of the skills. The results are shown in the table below.

VariantVerified Resolved (%)Rel. ΔPro Resolved (%)Rel. Δ
EvoGraph71.20–52.26–
w/o EHG66.00↓7.30%46.65↓10.73%
w/o Scope-Constrain70.00↓1.69%49.66↓4.98%
w/o History-Aware Maintenance66.80↓6.18%47.74↓8.65%
w/o Evolution History67.60↓5.06%48.70↓6.81%
w/o EHG Initialization69.60↓2.25%51.71↓1.05%

The results show that even with online skill updating, removing access to historical versions reduces the resolution rate by 5.06% and 6.81% on SWE-bench Verified and Pro, respectively. Compared with the frozen EHG (w/o History-Aware Maintenance), online updating alone recovers only 0.80 and 0.96 points, while historical information contributes about 80% of the total gain from history-aware maintenance on both benchmarks.

We have added this variant to Table 2 and Section 4.3 of the revised paper (highlighted in blue).

Q2 / W1: Compared with Related Works

According to your suggestion, we have clarified how EvoGraph differs from closely related methods, including CodeSkill, AutoSkill, Repo2Skill-Evo, and SkillHone.

Based on whether update decisions explicitly consult the prior evolution history of skills, we categorize existing approaches into history-ignoring and history-conditioned methods.

History-ignoring methods update skills based only on their current contents and newly acquired evidence. For example, AutoSkill [1] compares each candidate with its most similar existing skill to add, merge, or discard it; CodeSkill [2] learns an RL-based policy to manage skills from trajectory evidence and the current skill bank; Repo2Skill-Evo [3] evaluates updating a fixed skill set from a release patch. They overlook how each skill reached its current form, and thus cannot detect repeated additions or cumulative over-specialization across revisions. Different from them, EvoGraph retains every skill version in the EHG and assesses each update against its predecessors.

History-conditioned methods additionally consult records of earlier updates when revising skills. For example, SkillHone [4] records diagnoses, revisions, evaluation evidence, and outcomes to guide later revisions on practice probes. However, it evolves each skill in isolation and uses history only for revision. Different from it, EvoGraph: 1) organizes the history of all skills as a single graph, whose typed edges also capture relations across skills, rather than recording the history of each skill in isolation; 2) uses the evolution history for both retrieval and maintenance, online and without a separate evaluation suite, rather than only for revision; 3) further records repository scopes to constrain both retrieval and modification.

We have added this discussion as Section 2.3 of the revised paper (highlighted in blue).

References

[1] Yang, Yutao, et al. "Autoskill: Experience-driven lifelong learning via skill self-evolution." arXiv preprint arXiv:2603.01145 (2026).

[2] Li, Yanzhou, et al. "CODESKILL: Learning Self-Evolving Skills for Coding Agents." arXiv preprint arXiv:2605.25430 (2026).

[3] Duan, Chenyuan, et al. "Repo2Skill-Evo: Repository Skills Go Stale in Silence." arXiv preprint arXiv:2608.21964 (2026).

[4] Li, Zhiwei, and Yong Hu. "SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History." arXiv preprint arXiv:2606.08671 (2026).

AuthorsOpenReview ↗

Response to the Review [2/2]

Q3 / W3: Contribution of Evolution History to Reducing Knowledge Degeneration

To separate the effect of evolution history from that of the maintenance operations, we compare EvoGraph with the w/o Evolution History variant in Q1 (Qwen3.6-27B, SWE-bench Pro). Both variants start from the same initialized EHG and use the same operations and consolidation pass, differing only in access to historical versions.

StatisticEvoGraphw/o Evolution History
Active skills207242
Create actions144179
Skip actions174141
Total active skill length (K tokens)267.0300.9

Evolution history reduces active skills by 14.46% and total skill length by 11.27%. Since active skills increase only through creation and decrease only through consolidation, the gap in active skills (35) exactly matches the gap in create actions (35), indicating that consolidation contributes equally in both variants and the reduction stems from history-informed decisions to skip covered experience rather than create new skills.

We further manually inspect all create and update actions of both variants on the same 50 randomly sampled instances. A create action is redundant if its guidance is already covered by an existing skill or its earlier versions; an update action is over-specialized if it replaces general guidance with task-specific details or drops still-applicable guidance. Two authors annotate each action independently, with disagreements resolved by a third author (Cohen's κ = 0.910).

StatisticEvoGraphw/o Evolution History
Redundant create actions0 / 112 / 16
Over-specialized update actions3 / 3519 / 28

As shown in the table above, given the same maintenance operations, evolution history improves maintenance decisions and mitigates knowledge degeneration. We have added this analysis to Section 5 and Appendix E.2 of the revised paper (highlighted in blue).

Q4 / W4: Resolution–Cost Tradeoff

To evaluate the tradeoff between performance and inference cost, we compare the resolution rates of EvoGraph and the baselines with Qwen3.6-27B on SWE-bench Pro under per-task budgets of 1M, 2M, and 3M tokens. To ensure a fair comparison, we re-implement ReasoningBank on top of Claude Code, so that EvoGraph, Claude Code, and ReasoningBank + Claude Code share the same harness, and the cost of skill evolution can be separated from the overhead of the harness. We also report mini-SWE-agent as a reference. The results are shown in the table below.

MethodBudget 1M tokensBudget 2M tokensBudget 3M tokensNo budget capMean tokens without cap (M)
mini-SWE-agent14.6429.9639.2645.692.103
Claude Code14.9130.6438.0346.652.937
ReasoningBank + Claude Code15.0529.8236.8045.552.242
EvoGraph16.1433.6541.4551.713.389

The results show that the gains of EvoGraph do not come from its additional token consumption. Under the same scaffold, EvoGraph achieves the highest resolution rate across all budgets, with relative improvements of 8.25% to 9.82% over Claude Code and 7.24% to 12.84% over ReasoningBank + Claude Code. Moreover, as the budget increases from 1M to 3M tokens, the absolute gap over Claude Code widens from 1.23 to 3.42 points, indicating that EvoGraph converts additional tokens into resolved instances more effectively. Without a budget cap, EvoGraph consumes 15.39% more tokens than Claude Code on average, while improving the resolution rate by 10.85%. We consider this tradeoff acceptable for two reasons. First, EvoGraph remains the best method on the same harness when the budget is capped, so users can limit its cost without losing its advantage. Second, due to prompt caching, dollar cost does not grow linearly with token consumption, since about 92% of input tokens are cache hits, as measured in our experiments. Under the official API pricing of Qwen3.6-27B, EvoGraph costs $0.2977 per task on average, only 8.3% higher than the $0.2748 of Claude Code.

We have added budget curves with all three backbones on both benchmarks to Appendix E.4 of the revised paper (highlighted in blue). On the same harness, EvoGraph also outperforms Claude Code and ReasoningBank + Claude Code across budgets with these backbones.

Thanks for your efforts, the authors have addressed most of my concerns. However, I still have a question about why a graph is needed. If the authors can prove it clearly, I think I will maintain my original score.

Reviewer_ntYbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies knowledge duplication, overspecialization, and cross-repository contamination as coding agents continually accumulate experience, and proposes EvoGraph. The method uses skills as the medium for knowledge, employs an Evolution History Graph (EHG) to preserve skill versions and their revision, derivation, and merge relationships, and records whether their scope is general or repository-specific. Before task execution, the system combines semantic retrieval, graph expansion, and LLM routing to select skills; after task completion, it creates or revises skills based on execution traces, test feedback, and historical versions, and periodically merges overlapping skills. The authors implement the system on top of Claude Code and evaluate it with three backbone models on SWE-bench Verified and SWE-bench Pro, reporting improvements in resolution rates across six settings, along with analyses of knowledge base growth and cross-repository contamination.

Critical strengths
  1. The authors link history-blind updates to knowledge degradation and unclear scope to cross-repository contamination, and organize version history, provenance relationships, and repository scope within the same EHG. Historical information supports both retrieval and maintenance, while scope constrains both skill access and editing, forming a relatively complete continual learning workflow.
  2. EvoGraph achieves the highest resolution rate in all six settings across two benchmarks and three backbone models. The scope ablations in Table 2 and the analysis of contaminated calls in Section 5 also support the design of restricting repository scope. These results lead me to believe that the complete system has practical value for enabling coding agents to reuse experience.
Critical weaknesses
  1. The w/o History-Aware Maintenance setting in Section 4.3 actually freezes the initialized EHG, thereby removing both online experience accumulation and history-based maintenance. This comparison can demonstrate the effectiveness of continual maintenance, but cannot separately quantify the contribution of historical information.
  2. EvoGraph is based on Claude Code, whereas ReasoningBank is based on mini-SWE-agent, so differences in execution harnesses affect the comparison of knowledge management mechanisms. These issues limit the ability to determine the sources of the performance gains.
  3. Section 3.4 constrains edits through evidence citations and historical comparisons, but does not report how frequently capabilities regress after updates, or how much these constraints reduce detrimental modifications relative to standard updates.
  4. Section 5 and Table 4 primarily report the numbers of active skills and nodes, but do not yet report the length of individual skills, the size of the historical context, or how retrieval and maintenance overhead changes with the number of tasks.
Questions for authors
  1. Could at least one memory baseline be compared with EvoGraph using the same execution harness and backbone, with comparable initial experience and inference budgets?
  2. Does the current implementation validate new versions before they take effect, or reject or roll back updates when subsequent performance declines?
  3. How are the EHG and the embedding index actually stored and incrementally maintained?
3 comments
AuthorsOpenReview ↗

Response to the Review [1/3]

We thank the reviewer for the constructive feedback and the opportunity to clarify the following points:

Q1 / W2: Comparison with Memory Baselines under Matched Settings

To ensure a fair comparison, we compare EvoGraph with ReasoningBank using the same execution harness (Claude Code) and backbone (Qwen3.6-27B), with both methods starting without any initial experience. We conduct experiments both without a token budget cap and under a per-task budget of 3M tokens, based on the average per-task consumption of Claude Code on SWE-bench Pro without a cap (2.937M), as shown in the table below. The results under other budgets are shown as resolution-versus-budget curves in Figures 3 and 4 of the revised paper.

Method (Qwen3.6-27B, cold start)VerifiedVerified (3M)ProPro (3M)
Claude Code66.0063.2046.6538.03
ReasoningBank + Claude Code67.4053.4045.5536.80
EvoGraph69.6063.8051.7141.45

The results show that under fair comparison, EvoGraph consistently outperforms ReasoningBank, with relative improvements of 3.26% and 13.52% on SWE-bench Verified and Pro without a budget cap, and 19.48% and 12.64% under the 3M budget, respectively.

We have added this same-harness comparison under varying budgets with all three backbones to Appendix E.4 of the revised paper (highlighted in blue).

Q2: Skill Validation

EvoGraph targets the online deployment setting, in which a coding agent handles tasks one by one as they arrive and sees each task only once, while the repository keeps changing. Our evaluation follows this setting by processing tasks in chronological order.

Our current implementation does not validate new versions before they take effect, because such validation would incur high token and time costs. For example, validating a new version by rerunning one earlier task requires about 3M tokens and 29 minutes with Qwen3.6-27B on SWE-bench Pro (Table 1).

EvoGraph also does not reject or roll back updates when subsequent performance declines, because the repository keeps evolving and skills should stay consistent with its current state. For example, if a skill is updated to use a new interface introduced by a recent code change, rejecting or rolling back this update would revert the skill to guidance written for an earlier repository state, making it inconsistent with the current repository.

Our experimental results show that, even without such validation and rollback, EvoGraph still brings performance gains without a significant increase in cost.

Q3: Storage and Incremental Maintenance of the EHG and Embedding Index

The EHG is stored on the local file system in two parts. Skill contents, i.e., each SKILL.md file and its scripts, are stored under the SHA-256 digest of their files. The graph structure is stored as a JSON file that lists all nodes, including their skill identifiers, version numbers, content digests, routing descriptions, repository scopes, and activity statuses, together with all typed edges. Each update writes this file as a new snapshot, and a HEAD record points to the current one. Earlier snapshots are kept only for our analysis of skill evolution and are not required by EvoGraph.

After each task, EvoGraph compares each edited skill with the version it was copied from, and only newly created skills and skills whose contents have changed produce new versions (Appendix A.6). For each new version, its content is stored under its digest, and its corresponding node, together with its revision and derivation edges, is added to the graph file. The replaced version is only marked inactive, and its content is kept as history.

The embedding index is an on-disk cache that stores the Qwen3-Embedding-8B vector of each routing description, keyed by the digest of the text and the embedding model. Only new routing descriptions need to be embedded. Inactive versions keep their vectors, so that retrieval can still reach them and map them to their active replacements (Section 3.3). Therefore, neither the EHG nor the embedding index needs to be rebuilt after each task. We have added these implementation details to Appendix A.6 of the revised paper (highlighted in blue).

AuthorsOpenReview ↗

Response to the Review [2/3]

W1: Contribution of Evolution History

To isolate the contribution of historical information during skill updates, we construct an additional ablation variant, w/o Evolution History. This variant keeps continual skill maintenance but withholds historical versions and evolution relations from the skill editor. The results are shown in the table below.

VariantVerified Resolved (%)Rel. ΔPro Resolved (%)Rel. Δ
EvoGraph71.20–52.26–
w/o EHG66.00↓7.30%46.65↓10.73%
w/o Scope-Constrain70.00↓1.69%49.66↓4.98%
w/o History-Aware Maintenance66.80↓6.18%47.74↓8.65%
w/o Evolution History67.60↓5.06%48.70↓6.81%
w/o EHG Initialization69.60↓2.25%51.71↓1.05%

The results show that even with continual updates, removing evolution history reduces the resolution rate by 5.06% and 6.81% on SWE-bench Verified and Pro, respectively, accounting for about 80% of the total gain from history-aware maintenance on both benchmarks.

We have added this variant to Table 2 and Section 4.3 of the revised paper (highlighted in blue).

W3: Regression after Skill Updates and Effect of Editing Constraints

To measure how frequently capabilities regress after updates, we sample 30 skill updates from the skill evolution process of EvoGraph with Qwen3.6-27B on SWE-bench Pro. For each sampled update, the skill version before the update first helps resolve an earlier task, is later used on a different task that remains unresolved, and is then updated based on that failure. We rerun the earlier task with the updated version in place of the original one, keeping all other settings unchanged. Only 2 of the 30 reruns (6.67%) turn from resolved to unresolved, indicating that capability regression after skill updates is limited.


To measure how much the two constraints in Section 3.4 reduce detrimental modifications, we conduct two additional controlled analyses on the same 50 randomly sampled SWE-bench Pro instances. Specifically, historical comparison aims to prevent redundant create actions and over-specialized update actions, i.e., the two forms of knowledge degeneration, while evidence citation aims to prevent drifted actions, i.e., edits that are not supported by the experience of the task. For historical comparison, we use the w/o Evolution History variant in W1, which withholds historical versions and evolution relations from the skill editor. For evidence citation, i.e., requiring each atomic experience to cite supporting trajectory steps or repository code, we take the atomic experiences extracted by EvoGraph, remove these citations, and let the skill editor edit the same EHG again. We manually inspect all resulting create and update actions. A create action is redundant if its guidance is already covered by an existing skill or its earlier versions; an update action is over-specialized if it replaces general guidance with task-specific details or drops still-applicable guidance; an action is drifted if its guidance or condition is not supported by the trajectory steps and repository code of the task. Two authors annotate each action independently, with disagreements resolved by a third author (Cohen's κ = 0.910 for redundancy and over-specialization, and 0.822 for drift).

Detrimental modificationEvoGraphw/o Historical Comparisonw/o Evidence Citation
Redundant create actions0 / 112 / 16–
Over-specialized update actions3 / 3519 / 28–
Drifted actions1 / 46–5 / 51

The results show that historical comparison reduces redundant and over-specialized actions, while evidence citation reduces drifted actions, indicating that the two constraints prevent complementary types of detrimental modifications.

We have added both analyses to Appendix E.2 of the revised paper (highlighted in blue).

AuthorsOpenReview ↗

Response to the Review [3/3]

W4: Skill Length, Historical Context, and Overhead Scalability

We extend Table 4 with the mean length of active skills and the mean size of the historical context provided to the skill editor per task on SWE-bench Pro, as shown in the table below.

StatisticGPT-5 miniDeepSeek-V4-FlashQwen3.6-27B
Total nodes721747644
Active skills216241207
Revision edges489476433
Skip actions134166174
Mean active skill length (tokens/skill)1,1351,3291,290
Mean historical context (tokens/task)4,2674,4193,911

The results show that the historical context contains only 3,911–4,419 tokens per task, accounting for 0.23–0.29% of the maintenance overhead, so preserving evolution history adds negligible cost.


To analyze how retrieval and maintenance overhead changes with the number of tasks, we report their mean token consumption per task (M tokens), averaged over the first N processed tasks on SWE-bench Pro, as shown in the table below.

Processed tasks (N)GPT-5 mini RetrievalGPT-5 mini MaintenanceDeepSeek-V4-Flash RetrievalDeepSeek-V4-Flash MaintenanceQwen3.6-27B RetrievalQwen3.6-27B Maintenance
2000.021.410.041.920.061.30
4000.031.460.061.890.081.35
6000.061.480.071.920.091.37
7310.071.490.071.940.101.38

The results show that maintenance overhead remains stable as the EHG grows, increasing by at most 6.2% from 200 to 731 tasks. Although retrieval overhead grows with the number of skills, it remains at most 0.10M tokens per task, about 7% of the maintenance overhead. We have added the two statistics to Table 4 and the overhead analysis to Appendix E.3 of the revised paper (highlighted in blue).

Reviewer_G97nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoGraph, a history-aware and scope-constrained framework for continual skill evolution in coding agents. The framework is built upon an Evolution History Graph (EHG) that preserves historical skill versions, evolution relations, and repository scopes. Based on the EHG, EvoGraph employs scope-constrained skill retrieval and history-aware skill maintenance to mitigate knowledge degeneration and cross-repository contamination. Experiments on SWE-bench Verified and Pro across three backbone models report consistent improvements in task resolution rates over the evaluated baselines.

Critical strengths

valuable research problem. The paper addresses two challenges in continual skill evolution for coding agents: knowledge degeneration and cross-repository contamination.

Systematic framework design. EvoGraph organizes skill evolution history and repository scopes within a unified framework to support skill retrieval and maintenance.

Consistent empirical improvements. EvoGraph achieves the highest resolution rates among the evaluated methods across three backbone models on both SWE-bench Verified and SWE-bench Pro. The ablation studies and contamination analysis offer further support for the reported improvements.

Critical weaknesses

Limited generalizability and efficiency evaluation. The effectiveness of EvoGraph with stronger backbone models remains insufficiently explored, and the lack of runtime analysis makes it difficult to assess the practical cost of continual skill evolution.

Insufficient evaluation of knowledge degeneration. Although EvoGraph maintains fewer active knowledge items, the comparison involves knowledge representations of different granularities. More importantly, the evaluation lacks direct measurements of knowledge redundancy, over-specialization, and the preservation of previously useful knowledge, leaving the claimed mitigation of knowledge degeneration insufficiently validated.

Insufficient validation of the core mechanism. The current ablation study does not adequately isolate the contribution of history awareness from continual skill updating, leaving the effectiveness of a central mechanism insufficiently established.

Questions for authors
  1. In Figure 2, knowledge items refer to "active skills in EvoGraph, memory items in ReasoningBank, and saved tool scripts in Live-SWE-agent." These knowledge items may differ in granularity, making their counts difficult to compare directly. Moreover, inactive skill versions in EvoGraph can still participate in retrieval. Including them in the analysis would provide a more complete picture of knowledge-base growth.

  2. Table 1 suggests that the performance gains of EvoGraph tend to diminish as the backbone model becomes stronger. Evaluating EvoGraph with additional, more powerful backbone models would help further investigate this trend. Additionally, the paper only reports token consumption as a measure of computational cost. Other efficiency metrics, such as wall-clock time or execution speed, would provide a more comprehensive evaluation.

  3. In Section 4.3, the "w/o History-Aware Maintenance" variant freezes the EHG entirely, preventing continual skill updates. Therefore, this ablation does not isolate the contribution of historical information during updates. What would the performance be if skills were still continuously updated but without referring to historical versions? How much performance degradation would this introduce compared with the full EvoGraph?

4 comments
AuthorsOpenReview ↗

Response to the Review [1/3]

We thank the reviewer for the constructive feedback and the opportunity to clarify the following points:

Q1: Knowledge Granularity

To compare knowledge items at the same granularity, we re-implement the two baselines, i.e., ReasoningBank and Live-SWE-agent, with skills as the knowledge carrier, and conduct additional experiments on SWE-bench Pro. Specifically, the two skill-based variants retain the knowledge update mechanisms of the original methods, while generating and maintaining skills with the same prompts as EvoGraph, so that the skills of all three methods have the same granularity. The resolution rates and the number of knowledge items after processing the first 200, 400, 600, and all 731 tasks are shown in the tables below.

MethodGPT-5 miniDeepSeek-V4-FlashQwen3.6-27B
Live-SWE-agent (Skill)31.8745.2840.76
ReasoningBank (Skill)34.7548.0240.90
EvoGraph39.2652.8052.26
Method / #Processed tasks200400600731
GPT-5 mini
Live-SWE-agent (Skill)3168101122
ReasoningBank (Skill)6001,2001,8002,193
EvoGraph (all)266439610721
EvoGraph (active)117169206216
DeepSeek-V4-Flash
Live-SWE-agent (Skill)212431657800
ReasoningBank (Skill)6001,2001,8002,193
EvoGraph (all)275453633747
EvoGraph (active)119164228241
Qwen3.6-27B
Live-SWE-agent (Skill)107214328395
ReasoningBank (Skill)6001,2001,8002,193
EvoGraph (all)244397549644
EvoGraph (active)104153191207

The results show that EvoGraph achieves the highest resolution rate with all three backbones while maintaining 9.10–10.59× fewer active skills than ReasoningBank (Skill) and 1.91–3.32× fewer than Live-SWE-agent (Skill), except with GPT-5 mini, where Live-SWE-agent rarely saves knowledge (Section 5), indicating that the smaller knowledge base of EvoGraph comes from how skills are maintained rather than from the granularity of knowledge items.


In addition, we include inactive skill versions in the analysis, as shown by EvoGraph (all) in the table above. The results show that even with all historical versions counted, EvoGraph still contains 2.94–3.41× fewer knowledge items than ReasoningBank (Skill). Since inactive versions are always resolved to their active versions and never directly exposed to the coding agent, the knowledge base growth remains well controlled.

We have added the skill-based variants and the counts of all skill versions to Section 5 and Figure 2 of the revised paper (highlighted in blue).

AuthorsOpenReview ↗

Response to the Review [2/3]

Q2 / W1: Performance and Efficiency Analysis

The diminishing trend appears only on SWE-bench Verified, but not on SWE-bench Pro. Since SWE-bench Verified is approaching saturation, with OpenAI reporting that it no longer separates frontier coding agents well [1] (Section 4.2) and the strongest baseline with DeepSeek-V4-Flash already resolving 73.80% of its instances, the room for further improvement is limited. On the unsaturated SWE-bench Pro, the two stronger backbones yield the largest absolute gain over Claude Code, on which EvoGraph is built, as shown in the table below.

BackboneClaude CodeEvoGraphAbsolute gain
GPT-5 mini35.0239.26+4.24
Qwen3.6-27B46.6552.26+5.61
DeepSeek-V4-Flash47.1952.80+5.61

To further examine EvoGraph on frontier backbone models, we additionally conducted experiments with GPT-6-Luna, as shown in the table below.

MethodVerified Resolved (%)Verified ΔVerified #Token (M)Pro Resolved (%)Pro ΔPro #Token (M)
mini-SWE-agent75.60↑3.70%0.20247.88↑10.86%0.749
Claude Code75.80↑3.43%0.66648.42↑9.62%2.822
Live-SWE-agent74.80↑4.81%0.08247.33↑12.15%0.256
MemGovern74.80↑4.81%0.18348.42↑9.62%0.426
ReasoningBank73.00↑7.40%0.08146.65↑13.78%0.247
EvoGraph78.40–0.67953.08–2.827

The results show that the absolute gain over Claude Code on SWE-bench Pro remains substantial (4.66 points), within the 4.24–5.61 range of the other backbones. In addition, the performance gains of EvoGraph on SWE-Bench Pro do not diminish as the backbone model changes from GPT-5-mini (+4.24) to GPT-6-Luna (+4.66).


For a more comprehensive analysis of efficiency, we additionally report the average wall-clock time per instance, as shown in the table below.

Wall-Clock Time (min)Qwen3.6-27BGPT-5 miniDeepSeek-V4-FlashGPT-6-Luna
SWE-bench Verified
mini-SWE-agent17.741.9619.721.67
Claude Code17.137.2830.457.32
Live-SWE-agent14.242.6733.561.42
MemGovern14.094.5312.183.44
ReasoningBank9.255.1223.441.68
EvoGraph16.977.3729.876.34
SWE-bench Pro
mini-SWE-agent19.781.8016.854.37
Claude Code28.7021.4337.0521.12
Live-SWE-agent17.334.2231.293.14
MemGovern21.144.3919.045.24
ReasoningBank17.966.0924.803.68
EvoGraph29.0920.3236.1720.77

The results show that the wall-clock time of EvoGraph is comparable to that of Claude Code, on which it is built, and is even lower in 6 of the 8 settings, with at most 1.36% additional time in the remaining two. The longer time of EvoGraph relative to the other baselines therefore mainly comes from the underlying Claude Code harness. We have added the GPT-6-Luna results and the wall-clock time to Table 1, together with the corresponding discussion in Section 4.2 of the revised paper (highlighted in blue).

References

[1] OpenAI. Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities. 2026. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/

Q3 / W3: Contribution of Evolution History

To isolate the contribution of historical information during skill updates, we construct an additional ablation variant, w/o Evolution History, which keeps continual skill maintenance but withholds historical versions and evolution relations from the skill editor, and conduct additional ablation experiments, as shown in the table below.

VariantVerified Resolved (%)Rel. ΔPro Resolved (%)Rel. Δ
EvoGraph71.20–52.26–
w/o EHG66.00↓7.30%46.65↓10.73%
w/o Scope-Constrain70.00↓1.69%49.66↓4.98%
w/o History-Aware Maintenance66.80↓6.18%47.74↓8.65%
w/o Evolution History67.60↓5.06%48.70↓6.81%
w/o EHG Initialization69.60↓2.25%51.71↓1.05%

The results show that even with continual updates, removing evolution history reduces the resolution rate by 5.06% and 6.81% on SWE-bench Verified and Pro, respectively, accounting for about 80% of the total gain from history-aware maintenance on both benchmarks.

We have added this variant to Table 2 and Section 4.3 of the revised paper (highlighted in blue).

AuthorsOpenReview ↗

Response to the Review [3/3]

W2: Measurement of Knowledge Degeneration

For granularity, please see our response to Q1, where all methods use skills as the same knowledge carrier.


To directly measure knowledge degeneration, we manually inspect the create and update actions of EvoGraph and the w/o Evolution History variant in Q3 / W3 with Qwen3.6-27B on the same 50 randomly sampled SWE-bench Pro instances. A create action is regarded as redundant if its guidance is already covered by an existing skill or its earlier versions. An update action is regarded as over-specialized if it replaces general guidance with task-specific details or drops guidance that still applies to other tasks, which also covers the preservation of previously useful knowledge. Two authors annotated each action independently (Cohen's κ = 0.910), and disagreements were resolved by a third author. The results are shown in the table below.

StatisticEvoGraphw/o Evolution History
Create actions1116
Redundant create actions0 (0.00%)2 (12.50%)
Update actions3528
Over-specialized update actions3 (8.57%)19 (67.86%)

The results show that evolution history reduces redundant create actions from 12.50% to 0.00% and over-specialized update actions from 67.86% to 8.57%, indicating that evolution history mitigates knowledge degeneration.

We have added this analysis to Section 5 and Appendix E.2 of the revised paper (highlighted in blue).

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, with the deadline tomorrow, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear AC and Reviewers:

We thank the AC and all reviewers for their time and constructive feedback. Below, we summarize how we addressed the major concerns mentioned by the reviewers.

1. Validating the Contribution of Evolution History (Reviewers G97n, ntYb, UuKc)

To address the entanglement of evolution history and continual skill updating in the original ablation, we add a w/o Evolution History variant that keeps online updating but withholds historical versions, and conduct additional ablation experiments. Experimental results show that evolution history accounts for about 80% of the gain from history-aware maintenance (Section 4.3).

2. Extending the Analyses of Knowledge-Base Growth, Knowledge Degeneration, and Skill Update Reliability (Reviewers G97n, ntYb, UuKc)

To address the granularity mismatch and the lack of direct evidence on degeneration and update reliability, we have made the following efforts:

  • Knowledge-base growth at the same granularity. We re-implement the baselines with skills as the knowledge carrier and evaluate them with all three backbones. Experimental results show that EvoGraph performs best while maintaining 9.10–10.59× fewer active skills than ReasoningBank (Skill) (Section 5).

  • Knowledge degeneration. We manually annotate create and update actions with and without evolution history. The results show that evolution history eliminates redundant creations and reduces over-specialized updates from 67.86% to 8.57% (Section 5).

  • Skill update reliability. We ablate evidence citations and rerun previously resolved tasks with updated skills. Experimental results show that citations reduce unsupported edits and that only 6.67% of reruns regress (Appendix E.2).

3. Comparing Costs Along More Dimensions (Reviewers G97n, ntYb, UuKc)

To address practical cost, scalability, and the harness difference between methods, we have made the following efforts:

  • Wall-clock time and overhead scalability. We measure wall-clock time, the size of historical context, and per-task overhead as tasks accumulate. Experimental results show that EvoGraph is faster than Claude Code in 6 of 8 settings and that preserving evolution history adds negligible cost (Table 1, Appendix E.3).

  • Same-harness comparison under equal budgets. We re-implement ReasoningBank on Claude Code and compare the methods under equal token budgets. Experimental results show that EvoGraph performs best at every budget, outperforming ReasoningBank + Claude Code by up to 12.92% (Appendix E.4).

4. Clarified Novelty over Related Work (Reviewer UuKc)

We further discuss the distinction of EvoGraph from closely related methods (Section 2.3). Different from methods that update skills based only on their current contents, EvoGraph assesses each update against earlier versions to detect repeated additions and cumulative over-specialization. Different from methods that record the history of each skill in isolation and use it only for revision, EvoGraph organizes the history of all skills as a single graph and uses it for both retrieval and maintenance, constrained by repository scopes.

5. Evaluating with a Stronger Backbone (Reviewer G97n)

To address generalization to stronger backbones, we add GPT-6-Luna as a new backbone and evaluate all methods with it. Experimental results show that EvoGraph remains the best method and outperforms Claude Code by 4.66 points on SWE-bench Pro (Section 4.2).

Overall, these results further show that EvoGraph's improvements come from its evolution-history-aware skill maintenance rather than from knowledge-base size, computational cost, or the underlying harness. We would greatly appreciate it if the AC could take these new results, together with our detailed rebuttal and revised manuscript, into account in the final assessment.

Best regards,

The authors of submission 56287

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.