- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EviGate proposes a framework for self-evolving program-repair agents in which more persistent updates require broader evidence: public tests select patches, lessons pass through staged memory levels (L0-L2), and LoRA-trained models are kept only if they pass promotion, regression and safety gates, with every decision logged. On 20 Python tasks, HumanEval+ and SWE-bench, test-based best-of-N selection drives the main gain; memory and LoRA gating effects are small or inconclusive.
Critical strengths
- The problem is sound, solving a task is not necesarily correlated with learning a good behavior (reward hacking, getting lucky are common problems in the industry). Exploring research in this topic could advance SOTA practices for training agents for adquiring corret behaviors.
Critical weaknesses
- Not novel enough. The gates are basically held-out checkpoint selection plus regression tests, and rollback is just keeping the previous checkpoint.
- The main gain (+11 of 75 cases) comes from best-of-N selection by tests, which is already known (e.g., CodeT). So why do we need L0, L1, L2? Judges, memory staging and LoRA add little or nothing.
- Missing baseline: fine-tuning or RL on all passing trajectories with normal checkpoint selection. With a reasonable batch size the variance from lucky successes should mostly average out. If the authors' point is that reward hacking is a bias that doesn't average out, they need to show it.
- The gates don't work well: precision on harmful children is 0.25 and gate scores barely correlate with transfer (p = 0.095). They may add noise rather than remove it. On SWE-bench the full system solves 0 tasks and abstains on 99/100 (the baseline solves 5).
- The full L2 -> LoRA pipeline is never run end to end. The best LoRA result (+1.4) is post-hoc, and the pre-registered ones are inconclusive or negative.
- Very small setting: 20 Python tasks, 75 test cases, 5 training seeds. Maybe the problem is just too little data.
- Hard to read, too much jargon.
Questions for authors
- Could you treat test-time-scaling as baseline and show if this method is better than that?
- Do the gates still help with more tasks? Currently their transfer is insufficient
- Can you show the failures being gated are systematic (e.g., test hacking) and not just noise?
2 comments
We have not observed any revision commits from the authors, nor have we seen any improvements to the figures or tables. Therefore, we will maintain our score.
Thank you for suggesting direct comparisons with test-time scaling, ordinary fine-tuning, and wider validation.
EviGate's contribution is to turn evidence from a successful repair into a separately testable proposal for what future agents should inherit. In the controlled project follow-up, the same frozen model passed 16/42 private cases with five test-ranked rollouts and 25/42 with one rollout and an approved lesson. The parent weights were identical. The lesson carried a project convention learned on earlier tasks, and EviGate kept it even when the weight gate rolled back the adapters.
Q1. How does this compare with test-time scaling?
On the same candidate pool, public-test selection gained 11.05/75 cases over random selection. CodeT [1] also uses execution-based selection. In a nested-pool comparison with Qwen3.5-9B on 17 tasks and five seeds, Public N=1/3/5 passed 58.8/65.6/67.0 of 75 cases. Increasing N from 3 to 5 added 1.4 cases (paired seed×task 95% CI [−0.4, 3.8]) and raised mean completion tokens from 24.70k to 41.63k. On 41 report-only tasks derived from HumanEval+ Mini v0.1.10, the same budgets scored 204/204/202 of 220. The judge's mean gain over public-test selection was 0.42/75 (seed×task 95% CI [−0.17, 1.19]).
On the six report-only repairs in the 20-task project, the one-rollout memory arm solved 3/6 tasks and used 620 completion tokens at inference time. Frozen N=5 solved 0/6 and used 8,685. L0 recorded repair evidence, L1 checked cross-task support, and L2 committed the lesson for retrieval and training. That commit supplied four training targets. All five adapters completed training and weight validation. Four reached 31/42 on report-only cases despite being rolled back. This controlled experiment completed one inheritance cycle using executable tests. A workflow assistant could supply evidence at the same decision points through checks on task outcomes and action constraints (Appendix C).
In a separate 12-project ablation, requiring support from distinct source tasks blocked two harmful lessons and two helpful ones at the same evidence budget. Both policies scored 186/288 on 36 independent repairs.
Q2. Are public-passing errors systematic?
Across five seeds, 72/227 (31.7%) public-perfect candidate instances failed a private case. Malformed pointer handling and zero-capacity LRU behavior recurred across all three confirmation seeds. Serial replay verified these failures in five and nine distinct patches, respectively. An added diagnostic test found 0/17 suitable candidates among public-perfect patches in the frozen pools. Swapping a generic example for a targeted one at the same five-rollout budget left both arms at 6/36 on fresh edge cases. These experiments identify recurring behavioral failures.
SEAL [2] learns to generate self-edits for weight adaptation. Our ordinary fine-tuning control tests how much the agent can learn from the repairs it already produces. It used all 197 unique public-passing targets from 22 tasks and five seeds per context length. Ten targets failed extra source tests. At 1,024 tokens, ordinary validation selected a checkpoint with +7 aggregate validation cases that later fell from 95/95 to 88/95 on 11 independent related repairs. The strict gate preserved the parent. At 3,072 tokens all targets fit and both rules preserved the parent. This test of the averaging argument covers the expanded training pool and the two context lengths.
Q3. Does the gate improve with broader evidence?
On 72 repairs derived from MBPP+ v0.2.0 NoExtreme, increasing same-domain validation from 15 to 27 tasks (including three fixed safety tasks) left all ten children tied with the parent on validation cases. In an exploratory replay, increasing coverage from 192 to 747 cases on the same 24 tasks exposed losses for seven children. Six of them also lost cases on 48 independent tasks. This comparison identifies where more evidence made a measurable difference. Both retention rules kept the parent. The original ordinary-validation winner scored 315/384 on the independent cases versus 319/384 for the parent. Seven of ten children gained one solved task, so the case and task objectives give different judgments of rollback. The reported 0.095 is Spearman's ρ. Two of eight rejected adapters showed negative fresh-task transfer (precision 0.25).
For SWE-bench, the companion-code AST check had an inverted marker, which we corrected and verified in a Git fixture. Recovering the historical run version and candidate cache is necessary to attribute the reported abstentions. We will show the full L2 transaction, label Durable-role +1.4 as post-hoc, and simplify the figure and terminology.
References
[1] Chen et al. CodeT: Code Generation with Generated Tests. [ICLR 2023].
[2] Zweiger et al. Self-Adapting Language Models. [NeurIPS 2025].