acceptodds
Under review as a conference paper at ICLR 2027

EviGate: Evidence-Gated Inheritance for Self-Evolving Program-Repair Agents

Abstract

Large language model agents can generate patches, execute tests, and reuse repair trajectories, making program repair a natural setting for studying agents that adapt from their own interaction histories. Yet passing the evidence available on one task does not establish a reliable improvement: additional rollouts help only when their outcomes can be selected correctly, while local lessons may overfit visible tests and cause regressions when inherited through memory or parameters. We introduce EviGate, an evidence-gated inheritance framework that treats self-evolution as a sequence of separately authorized state transitions. EviGate combines execution-grounded candidate selection, staged experience consolidation, and parameter adaptation whose proposed children are retained only after promotion, regression, and safety checks, with append-only lineage connecting every decision. On 100 seeds and 17 fixed EvoCodeGym tasks, uninformed three-candidate sampling adds only private cases, whereas executable selection gains (seedtask 95% CI ). Memory, parameter, fresh-task, and repository-level evaluations further expose negative transfer and weak proposal coverage, establishing EviGate as an auditable inheritance protocol and empirical boundary map rather than a demonstration of repeated capability growth.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving agents, Program repair, Evidence-gated inheritance, Executable feedback

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZXNfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EviGate proposes a framework for self-evolving program-repair agents in which more persistent updates require broader evidence: public tests select patches, lessons pass through staged memory levels (L0-L2), and LoRA-trained models are kept only if they pass promotion, regression and safety gates, with every decision logged. On 20 Python tasks, HumanEval+ and SWE-bench, test-based best-of-N selection drives the main gain; memory and LoRA gating effects are small or inconclusive.

Critical strengths
  • The problem is sound, solving a task is not necesarily correlated with learning a good behavior (reward hacking, getting lucky are common problems in the industry). Exploring research in this topic could advance SOTA practices for training agents for adquiring corret behaviors.
Critical weaknesses
  • Not novel enough. The gates are basically held-out checkpoint selection plus regression tests, and rollback is just keeping the previous checkpoint.
  • The main gain (+11 of 75 cases) comes from best-of-N selection by tests, which is already known (e.g., CodeT). So why do we need L0, L1, L2? Judges, memory staging and LoRA add little or nothing.
  • Missing baseline: fine-tuning or RL on all passing trajectories with normal checkpoint selection. With a reasonable batch size the variance from lucky successes should mostly average out. If the authors' point is that reward hacking is a bias that doesn't average out, they need to show it.
  • The gates don't work well: precision on harmful children is 0.25 and gate scores barely correlate with transfer (p = 0.095). They may add noise rather than remove it. On SWE-bench the full system solves 0 tasks and abstains on 99/100 (the baseline solves 5).
  • The full L2 -> LoRA pipeline is never run end to end. The best LoRA result (+1.4) is post-hoc, and the pre-registered ones are inconclusive or negative.
  • Very small setting: 20 Python tasks, 75 test cases, 5 training seeds. Maybe the problem is just too little data.
  • Hard to read, too much jargon.
Questions for authors
  • Could you treat test-time-scaling as baseline and show if this method is better than that?
  • Do the gates still help with more tasks? Currently their transfer is insufficient
  • Can you show the failures being gated are systematic (e.g., test hacking) and not just noise?
2 comments

Thank you for suggesting direct comparisons with test-time scaling, ordinary fine-tuning, and wider validation.

EviGate's contribution is to turn evidence from a successful repair into a separately testable proposal for what future agents should inherit. In the controlled project follow-up, the same frozen model passed 16/42 private cases with five test-ranked rollouts and 25/42 with one rollout and an approved lesson. The parent weights were identical. The lesson carried a project convention learned on earlier tasks, and EviGate kept it even when the weight gate rolled back the adapters.

Q1. How does this compare with test-time scaling?

On the same candidate pool, public-test selection gained 11.05/75 cases over random selection. CodeT [1] also uses execution-based selection. In a nested-pool comparison with Qwen3.5-9B on 17 tasks and five seeds, Public N=1/3/5 passed 58.8/65.6/67.0 of 75 cases. Increasing N from 3 to 5 added 1.4 cases (paired seed×task 95% CI [−0.4, 3.8]) and raised mean completion tokens from 24.70k to 41.63k. On 41 report-only tasks derived from HumanEval+ Mini v0.1.10, the same budgets scored 204/204/202 of 220. The judge's mean gain over public-test selection was 0.42/75 (seed×task 95% CI [−0.17, 1.19]).

On the six report-only repairs in the 20-task project, the one-rollout memory arm solved 3/6 tasks and used 620 completion tokens at inference time. Frozen N=5 solved 0/6 and used 8,685. L0 recorded repair evidence, L1 checked cross-task support, and L2 committed the lesson for retrieval and training. That commit supplied four training targets. All five adapters completed training and weight validation. Four reached 31/42 on report-only cases despite being rolled back. This controlled experiment completed one inheritance cycle using executable tests. A workflow assistant could supply evidence at the same decision points through checks on task outcomes and action constraints (Appendix C).

In a separate 12-project ablation, requiring support from distinct source tasks blocked two harmful lessons and two helpful ones at the same evidence budget. Both policies scored 186/288 on 36 independent repairs.

Q2. Are public-passing errors systematic?

Across five seeds, 72/227 (31.7%) public-perfect candidate instances failed a private case. Malformed pointer handling and zero-capacity LRU behavior recurred across all three confirmation seeds. Serial replay verified these failures in five and nine distinct patches, respectively. An added diagnostic test found 0/17 suitable candidates among public-perfect patches in the frozen pools. Swapping a generic example for a targeted one at the same five-rollout budget left both arms at 6/36 on fresh edge cases. These experiments identify recurring behavioral failures.

SEAL [2] learns to generate self-edits for weight adaptation. Our ordinary fine-tuning control tests how much the agent can learn from the repairs it already produces. It used all 197 unique public-passing targets from 22 tasks and five seeds per context length. Ten targets failed extra source tests. At 1,024 tokens, ordinary validation selected a checkpoint with +7 aggregate validation cases that later fell from 95/95 to 88/95 on 11 independent related repairs. The strict gate preserved the parent. At 3,072 tokens all targets fit and both rules preserved the parent. This test of the averaging argument covers the expanded training pool and the two context lengths.

Q3. Does the gate improve with broader evidence?

On 72 repairs derived from MBPP+ v0.2.0 NoExtreme, increasing same-domain validation from 15 to 27 tasks (including three fixed safety tasks) left all ten children tied with the parent on validation cases. In an exploratory replay, increasing coverage from 192 to 747 cases on the same 24 tasks exposed losses for seven children. Six of them also lost cases on 48 independent tasks. This comparison identifies where more evidence made a measurable difference. Both retention rules kept the parent. The original ordinary-validation winner scored 315/384 on the independent cases versus 319/384 for the parent. Seven of ten children gained one solved task, so the case and task objectives give different judgments of rollback. The reported 0.095 is Spearman's ρ. Two of eight rejected adapters showed negative fresh-task transfer (precision 0.25).

For SWE-bench, the companion-code AST check had an inverted marker, which we corrected and verified in a Git fixture. Recovering the historical run version and candidate cache is necessary to attribute the reported abstentions. We will show the full L2 transaction, label Durable-role +1.4 as post-hoc, and simplify the figure and terminology.

References

[1] Chen et al. CodeT: Code Generation with Generated Tests. [ICLR 2023].

[2] Zweiger et al. Self-Adapting Language Models. [NeurIPS 2025].

We have not observed any revision commits from the authors, nor have we seen any improvements to the figures or tables. Therefore, we will maintain our score.

Reviewer_GA6COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviGate, an evidence-gated inheritance framework for self-evolving program-repair agents. The framework integrates execution-guided candidate selection, staged experience consolidation, and parameter adaptation with promotion, regression, and safety gates. Experiments on EvoCodeGym and external benchmarks investigate the effectiveness of these components and their ability to prevent harmful updates.

Critical strengths

1.The paper provides relatively comprehensive component-level evaluations and reports both positive and negative experimental results. 2.The paper addresses the problem of preventing unreliable improvements from being inherited by self-evolving agents.

Critical weaknesses
  1. The novelty is limited. The proposed method mainly consists of stacking existing modules and lacks core innovations. Meanwhile, the proposed method is largely heuristic and lacks theoretical justification.
  2. Section 5.5 says that "The primary Raw- and Filtered-public controls do not instantiate the full L2 pathway." However, staged Evidence-Gated Inheritance is precisely the central contribution of this paper. The main experiments do not implement the complete proposed framework, and the multi-cycle experiment produces no durable experience or parameter updates, failing to empirically validate its own contribution.
  3. The framework figure and experimental tables exhibit noticeable AI-assisted or automatically generated-looking formatting, and the writing appears very defensive.
Questions for authors

see weakness.

2 comments

Thank you for asking us to clarify the framework's contribution and test the complete inheritance path.

EviGate specifies when a successful repair can change what an agent remembers and which weights it deploys. In a controlled run of the full pipeline, a lesson learned from independent public repairs raised performance from 11/42 to 25/42 private cases on unseen repairs. The framework retained that lesson through training and rolled back the proposed adapters independently.

Q1. Design contribution and rationale

EviGate gives each persistent change a reference version and an explicit acceptance rule. L0 records evidence from a repair, L1 accumulates support across tasks, and L2 commits validated lessons for retrieval and training. Let θ and M denote the original weights and memory, M′ the approved memory, and θ′ the candidate weights. For evaluation score Q, the joint change decomposes into Q(θ,M′) − Q(θ,M) for memory and Q(θ′,M′) − Q(θ,M′) for the adapter. Keeping M′ fixed measures what the adapter adds to the agent that would deploy it. An adapter rollback then preserves the approved lesson. ACE [1] evolves reusable context from execution feedback on AppWorld. A data-analysis agent could supply evidence for EviGate through reference queries and checks on approved reports (Appendix C).

This distinction changed real decisions in a post-hoc replay of the controlled project. Memory alone added three promotion cases. Using the same gate, a joint comparison to the original parent admitted 3/5 adapters, while evaluating against the parent with approved memory admitted 0/5. The independent endpoint also showed a complementary effect. None of the five adapters gained cases with retrieval off, while four gained six cases with retrieval on. The fixed reference isolates the adapter's contribution on the measured tasks. The interaction on unseen tasks motivates stronger coverage for that comparison.

Q2. Evidence for the complete path and its components

In the 20-task controlled project, public repairs on three distinct source tasks supported a shared tie-breaking convention. The L2 commit froze the first two supporting tasks and supplied four deduplicated training targets. On 13 validation tasks, the lesson gained one promotion case and two solved selection tasks while passing regression and safety checks. We trained five L2 adapters and five controls with the same target patches and the lesson omitted from the control prompts. The submitted frozen follow-up stopped before L2. In the new run, all five L2 adapters reached the weight gate and were rolled back with zero promotion gain. Two also failed safety checks. The retained version solved 3/6 independent repairs, versus 0/6 for the frozen parent. Four rolled-back L2 adapters and one passing matched-target control reached 31/42 private cases. This controlled cycle traces a committed lesson from source repairs through training and a separate weight decision.

An equal-budget ablation across 12 projects isolates the effect of cross-task support. Given the same candidate lessons and final validation, two checks on distinct source functions blocked two harmful lessons that two checks on one function would retain. Two beneficial lessons were also blocked, leaving both policies at 186/288 on 36 unseen repairs. A batch rule with the same support evidence reproduced the decisions. On 54 repairs derived from HumanEval+ Mini v0.1.10, one-step validation gained two of 95 report-only cases and the staged rule retained the parent. These experiments show how an explicit support requirement changes retention under a fixed validation budget.

The ordinary SFT control used all 197 unique public-passing targets from 22 tasks. At 1,024 tokens, its selected checkpoint gained seven aggregate validation cases but lost seven of 95 related test cases. The strict gate preserved the parent. At 3,072 tokens all targets fit and both policies preserved the parent. The 72-task study derived from MBPP+ v0.2.0 NoExtreme also favored rollback under either rule on the measured validation criteria. On 48 independent repairs, one of ten adapters improved the total cases passed, and seven solved one extra task. We will report both objectives. Section 5.5 will present the complete transaction first, with Raw/Filtered labeled as component controls and Durable-role as post-hoc.

Q3. Figure, tables, and writing

We will redraw Fig. 1 to show how EviGate chooses a patch, retains a lesson, and replaces a model. The tables will identify the baseline, denominator, and split for each result. We will explain the memory states once using the controlled run as an example. The submission discloses assistance with language polishing and figure design.

References

[1] Zhang et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. [ICLR 2026].

We have not observed any revision commits from the authors, nor have we seen any improvements to the figures or tables. Therefore, we will maintain our score.

Reviewer_o7ZfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how program-repair agents decide what to retain from their repair attempts. A fixed execution pipeline generates candidate repairs and selects an answer using public tests. Proposed lessons are staged before becoming reusable guidance, while LoRA updates undergo separate improvement, regression, and safety checks before replacing the parent model. Experiments use shared candidate pools to isolate selection effects, then examine memory retention, parameter adaptation, and transfer to new tasks and repositories. Execution-based selection provides the clearest gains. Staged memory reduces some observed negative transfer, but parameter benefits are inconsistent across evaluation settings, and repository-level evaluation produces extensive abstention. The studies evaluate individual components rather than demonstrate a complete, repeatedly improving agent.

Critical strengths
  1. The research question is practically important. Experience that helps an agent succeed on one task may not justify incorporating it into persistent memory or model parameters. Once inherited, an incorrect lesson can affect many subsequent tasks. Understanding what evidence is sufficient to support these persistent updates is an important problem for agents that learn from their own interactions.

  2. The paper is transparent about its results and limitations. The authors report several unfavorable outcomes, including the gates’ weak predictive value on fresh tasks, extensive abstention in repository-level evaluation, and the full-cycle follow-up stopping before parameter adaptation. They also explicitly label post-hoc analyses and appropriately limit their claims.

Critical weaknesses
  1. The paper provides limited new information. Public tests can help select better patches, accumulated experience can harm subsequent tasks, and completing training does not establish that a model has improved. These observations are already familiar, and GRASP [1] already uses held-out validation and a hard regression budget to decide whether to accept experience updates. This paper separately manages task-level selection, experience retention, and parameter updates, but does not clearly explain what previously unresolved problem this separation addresses. The clearest experimental gain still comes from selecting patches using execution results. Even if these results are reliable, they offer limited new understanding of how agents can improve over time.

  2. The paper does not clearly establish why its multi-stage experience and parameter-update process is needed. The authors emphasize that only experience passing through L₀/L₁/L₂ can enter training, yet the Raw/Filtered parameter experiments bypass this process, Durable-role is a post-hoc approximation, and the full-cycle experiment never completes a parameter update. These experiments therefore do not directly show what is gained by progressively accumulating and validating experience. The paper also lacks a comparison with simply evaluating an updated agent and retaining the original version when performance regresses. Consequently, it remains unclear which benefits come from the proposed staged design and which could already be achieved through ordinary validation and rollback.

  3. The experiments do not clearly demonstrate that the gates can identify updates that benefit subsequent tasks. In the memory study, the reported reduction in negative transfer partly comes from tasks directly checked by the gate, while the independent evaluation endpoint shows no improvement. The fresh-task evaluation likewise does not establish that gate-approved models are better choices for retention. The reasons for these limited results also remain unclear. On SWE-bench, only 12 of 498 candidates pass all hard gates, so the high abstention rate could reflect unusable proposals, overly strict checks, or both. Fresh-task performance could similarly depend on inadequate validation coverage or limited transferability of the updates. The analysis does not sufficiently separate these factors, making it difficult to assess either the gates’ effectiveness or the main source of their failures.

[1] GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents.

Questions for authors

See Weaknesses.

2 comments

Thank you for asking what each inheritance check adds beyond conventional validation.

EviGate treats patch selection, lesson retention, and weight replacement as distinct decisions. In a controlled project, an approved lesson raised performance from 11/42 to 25/42 private cases on six unseen repairs. EviGate preserved that gain by keeping the lesson while rolling back all five weight updates.

Q1. What does this add to GRASP?

GRASP [1] applies held-out validation and a regression budget to skill-library edits. EviGate links the source repairs, retained lesson, training targets, and deployment decision in a version trace. Once a lesson passes the memory gate, the weight gate compares the parent and child with the same approved memory. In a post-hoc replay using the same tests and gate, evaluating memory and weights together against the original parent admitted 3/5 adapters. With memory fixed, the gate admitted 0/5. The three-case promotion gain was already present with memory alone. This illustrates why evidence sufficient to retain one update can be insufficient to retain another. GEPA [2] evolves prompts, and the Darwin Gödel Machine [3] evolves the agent implementation. The same inheritance workflow could govern updates in either setting using independent evaluation and regression checks (Appendix C).

Q2. What does staging add to ordinary validation?

Across 12 controlled projects, we fixed the candidate lessons and final validation rule and spent the same two checks on either one source function or two. On 36 independent repairs, two-source support blocked both harmful lessons that ordinary validation retained. It also blocked two helpful lessons when the second function was a local exception. Both policies passed 186/288 cases (paired project-bootstrap 95% CI for the difference [−60, 60]). A batch rule given the same independent evidence reproduced the decisions. The experiment identifies the value and cost of requiring independent support. Staging makes that requirement explicit for an agent receiving experience over time. On 54 repairs derived from HumanEval+ Mini v0.1.10, one-step validation scored 93/95 on independent cases versus 91/95 for staging, with 10/11 tasks solved by both.

For weights, we compared the gate with ordinary checkpoint selection after LoRA on 197 unique public-passing repairs from 22 tasks, using five seeds at each training length. With 1,024 tokens, the ordinary selection winner gained seven aggregate validation cases, then fell from 95/95 to 88/95 on 11 independent related repairs. It tied the parent on 41 additional HumanEval+–derived tasks. The strict gate preserved the parent. At 3,072 tokens all targets fit and both rules kept the parent. The controlled project above also completed the L2-to-LoRA path using four targets from a committed lesson. Four rolled-back adapters reached 31/42 on the independent endpoint, so the weight gate also missed possible gains. We will distinguish this run from the Raw/Filtered component controls and the post-hoc Durable-role proxy.

Q3. Which evidence helps predict transfer?

On 72 repairs derived from MBPP+ v0.2.0 NoExtreme, adding cases to the same 24 validation tasks in an exploratory replay (192 to 747) exposed losses for seven checkpoints. Six of those seven also lost cases on 48 separate repairs. Increasing the number of same-domain validation tasks from 15 to 27 had left every checkpoint tied. Both rules kept the parent. On the 24-task same-domain panel, a rollback rule that accepted ties would have admitted all ten adapters. On the independent repairs, seven lost cases and seven solved one additional task. The original ordinary-validation winner scored 315/384 there, versus 319/384 for the parent. In the submitted memory study, staging preserved the predefined memory-independent selection endpoint. The original adapter transfer estimate was Spearman ρ=0.095, with two harmful updates among eight rejected.

For SWE-bench, we reproduced and corrected an inverted AST-change check in the companion code. The fix passed the substantive change in a Git fixture. Separating unusable proposals from gate rejections in the historical run requires verifying its code version and replaying its cached candidates.

References

[1] Moll et al. GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents. [EMNLP 2026].

[2] Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. [ICLR 2026].

[3] Zhang et al. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. [ICLR 2026].

Thank you for the response and additional experiments, which have addressed some of my concerns. However, staged lesson retention and separate evaluation of weight updates still seem somewhat incremental to me. I will therefore maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.