acceptodds
Under review as a conference paper at ICLR 2027

SelfSearch: Reward-Free Search for Self-Improving Agents

Abstract

Advances in the coding capabilities of LLM agents allow them to inspect and modify their own instructions, tools, and execution procedures. Existing approaches use this ability to search for improved agents through repeated downstream evaluation, which incurs substantial costs and ties the search to the evaluated tasks. We introduce SelfSearch, a reward-free search procedure in which agents modify themselves using records of previous self-improvement episodes. These records capture the reasoning, tool actions, and outcomes of earlier modification attempts, providing concrete experience for improving both task solving and self-modification. Without downstream reward signals during search, SelfSearch discovers agents that improve success rates by up to 11.2 percentage points on Terminal-Bench 2.1, 6.7 on SWE-bench Multilingual, and 5.0 on SWE-bench Verified. On SWE-bench Multilingual, an agent improves success by 5.0 percentage points while reducing execution cost by 38.5% on tasks solved by both the initial and evolved agents. SelfSearch achieves competitive task success with evaluation-guided search baselines at lower search cost. With only $4.03 in search cost, it produces a harness achieving 82.0% on Terminal-Bench 2.1 using DeepSeek V4 Flash, comparable to the reported performance of Codex, the top-performing harness in a published nine-harness comparison. These results suggest that experience gained through self-modification can improve agents' downstream capabilities and efficiency.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
self-improving agents, recursive self-improvement, agent harnesses, reward-free search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ftfFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

SelfSearch improves agent code using records of earlier self-modification episodes, without downstream benchmark feedback during search. Model weights stay fixed. Two agent lineages share records, and each updated agent performs the next editing episode. The authors test the final agents on SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.1. They report higher average success across the two lineages in all six model–benchmark settings, with some cost reductions and cross-model transfer. Ablations and case studies examine the roles of past records, the evolving improver, and the tools it develops.

Critical strengths
  1. Using self-editing experience to improve agents is a useful idea. The method avoids repeated downstream evaluation during search. Reporting both final lineages also avoids choosing the main results based on downstream checkpoint scores.

  2. The results cover two model settings and three benchmarks. The improved harnesses transfer across model families without further search. Comparing costs on tasks solved by both agents also gives useful evidence of execution efficiency.

  3. The case studies show practical changes, such as keeping matching text in search results and linking tool calls to their results across trace pages. The ablations also test two relevant choices: access to past records and whether the improver changes.

Critical weaknesses
  1. The initial agent lacks common tools such as exact-text replacement and line-range viewing (Appendix A). It is unclear how much of the gain comes from adding these tools. The fixed-improver ablation provides some support for recursive updating, but only from this limited starting point. The paper does not show whether the benefit remains when both the initial agent and the fixed improver already have these tools.

  2. Each model setting has one search with two lineages that share records. These are not independent repeats. Without repeated searches or error bars, it is hard to judge how stable the gains are, especially the 1.7–2.9 percentage-point differences in the fixed-improver ablation.

  3. Related work and Table 13 omit Gödel Agent (Yin et al., ACL 2025), which already allows an agent to change both its task policy and its improvement logic. The paper should explain this difference and compare what each method can modify and when those changes take effect.

  4. The discovered changes could provide clearer lessons for harness design. Tables 5 and 11 list changes, while Table 12 counts tool use without showing how much each tool helps.

  5. The comparisons do not fully separate method gains from budget differences. The evaluation-guided baselines use different search budgets and selection rules (Appendix C.3). The 82.0% Terminal-Bench result also uses higher reasoning effort and larger execution limits, without reporting the initial agent's performance under the same settings. These results do not show the best method at equal cost or SelfSearch's contribution to that high-resource result.

Questions for authors
  1. Could the authors start with the common editing and search tools already installed, then compare no search, a fixed improver, and an evolving improver under the same budget? This would directly test the added value of recursive updating.

  2. Could the authors repeat the search and report performance across generations? Intermediate agents can be evaluated after search ends, so those results do not guide search. This would show whether gains are stable and continue beyond the early tool additions.

  3. Could the authors expand the search-excerpt and trajectory-reader examples to show the original failure, the code change, and the effect on errors, calls, or cost? Examples where a change does not help would also make the design lessons more useful.

7 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/5)

We thank the reviewer for the constructive feedback and suggestions for additional controls, quantitative analysis, and clearer design lessons.

W1. The initial agent lacks common tools such as exact-text replacement and line-range viewing (Appendix A). It is unclear how much of the gain comes from adding these tools. The fixed-improver ablation provides some support for recursive updating, but only from this limited starting point. The paper does not show whether the benefit remains when both the initial agent and the fixed improver already have these tools.

Q1. Could the authors start with the common editing and search tools already installed, then compare no search, a fixed improver, and an evolving improver under the same budget? This would directly test the added value of recursive updating.

We started from an initial agent already equipped with exact-text replacement, line-range viewing, and bounded search. We compared no search, a fixed initial improver, and SelfSearch. The fixed-improver method used the same initial agent as both the frozen editor and the starting target, with the same per-generation editing budget as SelfSearch. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna.

Comparison with a fixed improver.

MethodCapabilityAdaptive
Initial agent (standard tools)76.6776.67
Fixed initial improver (standard tools)75.8375.83
SelfSearch (standard tools)76.6776.11

Values are SWE-bench Verified accuracy (%) on the same 120 tasks. The initial-agent score averages three evaluations. Fixed-initial results come from one search, while SelfSearch results average three independent searches.

With standard tools already available, SelfSearch scored slightly higher than the fixed improver on SWE-bench Verified, while remaining close to the initial agent’s performance.

Evaluation on Terminal-Bench. We also evaluated the final SelfSearch agents on Terminal-Bench to examine whether improvements from this starting point extend to a broader set of terminal tasks.

AgentAccuracy (%)
Initial agent (standard tools)53.93
Capability (standard tools)67.42
Adaptive (standard tools)66.29

All searches started from an initial agent equipped with standard editing and search tools. All agents were evaluated on the same 89 Terminal-Bench 2.1 tasks. Capability and adaptive scores average the final agents from three independent searches. The initial-agent score comes from one evaluation.

Both branches exceeded the initial agent in all three searches, with mean gains of 13.48 and 12.36 percentage points. These results show that improvements remain possible when standard editing and search tools are already present, although the gains differ across benchmarks.

The improvements included repairs to execution behavior. The initial agent’s shell mishandled valid multiline commands, causing repeated tool timeouts. The evolved agents repaired this behavior. For example, on the Terminal-Bench task count-dataset-tokens, all four multiline commands attempted by the initial agent timed out. The Search 2 capability agent executed eight multiline commands without a timeout and solved the task. This illustrates how self-editing can improve the reliability of an agent that already has standard tools.

AuthorsOpenReview ↗

Official Comment by Authors (2/5)

W2. Each model setting has one search with two lineages that share records. These are not independent repeats. Without repeated searches or error bars, it is hard to judge how stable the gains are, especially the 1.7–2.9 percentage-point differences in the fixed-improver ablation.

Starting from the original initial agent, we conducted three independent searches each for SelfSearch and the fixed initial improver. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna.

MethodCapabilityAdaptive
Fixed initial improver75.83 ± 3.0075.83 ± 1.67
SelfSearch80.00 ± 1.6777.50 ± 2.89

Values are mean ± sample standard deviation of SWE-bench Verified accuracy (%) on the same 120 tasks. Fixed-initial and SelfSearch results each summarize three independent searches.

Full SelfSearch outperformed the fixed improver by 4.17 percentage points for the capability branch and 1.67 percentage points for the adaptive branch. These results support the benefit of allowing the improver to evolve during search.

W3. Related work and Table 13 omit Gödel Agent (Yin et al., ACL 2025), which already allows an agent to change both its task policy and its improvement logic. The paper should explain this difference and compare what each method can modify and when those changes take effect.

Gödel Agent is closely related to our work, as it can modify both its task-solving policy and improvement logic. We have added it to the related-work discussion and Table 13, and expanded Appendix F to clarify what each method modifies and when changes take effect. Gödel Agent uses runtime monkey patching, while SelfSearch edits repository code that is executed in the next agent episode and carries forward the records of making those edits. A central distinction is the feedback used during search. Gödel Agent’s reported experiments use validation-task performance to guide improvement, whereas SelfSearch uses self-editing records and local checks without downstream evaluation rewards.

AuthorsOpenReview ↗

Official Comment by Authors (3/5)

W5. The comparisons do not fully separate method gains from budget differences. The evaluation-guided baselines use different search budgets and selection rules (Appendix C.3). The 82.0% Terminal-Bench result also uses higher reasoning effort and larger execution limits, without reporting the initial agent's performance under the same settings. These results do not show the best method at equal cost or SelfSearch's contribution to that high-resource result.

Search costs and feedback comparison. Our original comparison evaluates complete search procedures under matched editing configurations, with search costs reported explicitly. Linear and archive search incurred total search costs of \12.35and12.35 and \\7.53 with GPT, and \8.59and8.59 and \\7.90 with DeepSeek, respectively, compared with \6.52and6.52 and \\4.03 for SelfSearch. These costs include the development evaluations used to guide the baselines, while SelfSearch requires no downstream evaluations during search. SelfSearch therefore achieved competitive performance at lower recorded search cost, despite differences in search structure and selection rules.

To examine the role of feedback while retaining the search structure, we conducted an additional comparison using the same initial agent, two lineages, ten editing generations, and per-edit execution limits. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna. This comparison contrasts experience from solving downstream tasks with experience from modifying the agent’s own tools, policies, and execution logic. We replaced inherited self-editing trajectories with task-solving trajectories, submitted patches, and evaluation outcomes. Before each edit, each lineage was evaluated on one task sampled from a ten-task development pool, and both records were shared with both lineages. Development scores did not select or reject candidates.

Experience sourceCapabilityAdaptive
Self-editing experience79.7077.58
Task-solving evaluation experience76.3675.45

Values are mean SWE-bench Verified accuracy (%) across three independent searches for each method, evaluated on the same 110 tasks excluding the ten development tasks. The SelfSearch scores use the same three searches reported above, restricted to these 110 tasks.

Self-editing experience yielded higher mean accuracy by 3.33 and 2.12 percentage points for the capability and adaptive branches, respectively. Both methods had the same editing opportunities, while the task-feedback method incurred additional development-evaluation costs. These results suggest that self-editing trajectories can provide more useful guidance for agent improvement under this search structure.

Correction to the Terminal-Bench comparison. For the original high-resource Terminal-Bench comparison, we requested DeepSeek V4 Flash through OpenRouter, but the inspected OpenRouter response records identified the served model as DeepSeek V4.1 Flash. We have corrected the manuscript and withdrawn the original claim of a model-matched comparison with the published Codex result.

In a follow-up evaluation using DeepSeek V4 Flash at max reasoning effort, SelfSearch solved 70 of 89 tasks (78.7%), compared with 66 (74.2%) for the initial agent. This score would rank second among the scores reported in the nine-harness comparison, behind Codex.

Reviewer_tKHkOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SelfSearch iteratively updates an agent's codebase using logs from past self-modification attempts, without relying on downstream benchmark evaluations during search. With model weights frozen, two agent lineages evolve in parallel by exchanging modification trajectories, code diffs, and local test results. Tested across two models and three coding benchmarks, the method improves average performance, efficiency, and cross-model transfer compared to evaluation-guided baselines and ablations.

Critical strengths

Exploring how far agents can improve without external benchmark feedback is a compelling direction that could reduce reliance on costly task-specific evaluations. I like the idea that self-modification alone can yield transferable improvements, and that final-generation agents can do so directly, without selecting checkpoints with downstream feedback and grading.

Critical weaknesses

Incremental novelty and narrow practical scope

Core components like self-modification, experience accumulation, and evolving meta-agents are already established in prior work (e.g., DGM, Hyperagents, Meta-Harness). In fact, Table 13 notes that SICA shares nearly all of SelfSearch's properties except reward-free search. Because the method itself is a modest extension, the paper's value rests heavily on what happens under this reward-free regime.

Unfortunately, the actual modifications are mostly basic tooling fixes. The evolved improvements consist mainly of standard features like exact-text replacement, file viewing, output truncation, and whitespace fixes (Tables 5 & 11). It is unclear if SelfSearch can drive meaningful progress once an agent already has basic and standard tooling.

Weak mechanistic evidence

The paper presents an indirect test of the evolving improver. The fixed-improver ablation only measures final downstream scores, not whether later-generation agents are actually better at editing given the same starting point and budget. Without performance curves across generations, it looks like early generations simply patch obvious tooling gaps while later ones yield diminishing returns.

The performance improvements lack statistical rigor and fair baselines. The reported gains in Table 4 lack error bars or independent runs. Additionally, the evaluation-guided baselines change both the feedback signal and search structure simultaneously, and their comparison are not controlled within the same budget.

Questions for authors
  • Can you run an ablation that keeps cross-lineage code snapshots and diffs while stripping out the step-by-step editing trajectories? This would isolate the value of the modification process from simple code sharing.

  • Do the gains hold if the base agent starts with standard tools (e.g., exact-text replacement, line-range viewing, bounded search)? Sharing performance across intermediate generations would also clarify if progress stalls after basic tools are added.

  • Can you compare early- and late-generation improvers on the exact same starting codebase and budget to directly test whether the agent's ability to self-improve actually gets better?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the detailed assessment and suggestions for strengthening the experimental controls and clarifying the contribution.

W1. Incremental novelty and narrow practical scope

Core components like self-modification, experience accumulation, and evolving meta-agents are already established in prior work (e.g., DGM, Hyperagents, Meta-Harness). In fact, Table 13 notes that SICA shares nearly all of SelfSearch's properties except reward-free search. Because the method itself is a modest extension, the paper's value rests heavily on what happens under this reward-free regime.

Unfortunately, the actual modifications are mostly basic tooling fixes. The evolved improvements consist mainly of standard features like exact-text replacement, file viewing, output truncation, and whitespace fixes (Tables 5 & 11). It is unclear if SelfSearch can drive meaningful progress once an agent already has basic and standard tooling.

Q2. Do the gains hold if the base agent starts with standard tools (e.g., exact-text replacement, line-range viewing, bounded search)? Sharing performance across intermediate generations would also clarify if progress stalls after basic tools are added.

SelfSearch investigates whether useful agent improvements can be discovered without downstream evaluation rewards during search. Its central mechanism is to retain experience from modifying the agent’s own tools, policies, and execution logic, and use that experience to guide subsequent revisions. The contribution lies in discovering useful agent revisions without downstream evaluation rewards, even when the resulting features are familiar. We have updated Table 13 to clarify the use of self-editing trajectories. To test whether the benefits extend beyond adding missing standard tools, we conducted the following experiments.

Starting with standard tools. We conducted three independent searches starting from an agent already equipped with exact-text replacement, line-range viewing, and bounded search. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna.

BenchmarkInitial agentCapabilityAdaptive
SWE-bench Verified76.6776.6776.11
Terminal-Bench 2.153.9367.4266.29

Values are accuracy (%). Final-agent scores are averaged across three independent searches. The initial-agent score averages three evaluations on SWE-bench Verified and uses one evaluation on Terminal-Bench.

On Terminal-Bench, both branches improved over the initial agent in all three searches, with mean gains of 13.48 and 12.36 percentage points. SWE-bench performance remained close to the initial level. These results show that standard tools do not eliminate the opportunity for improvement, although the gains depend on the benchmark.

Providing standard editing and search features does not ensure that the agent’s execution machinery works reliably in practice. Self-editing exposes these operational problems while the agent uses and revises its own implementation. The initial agent’s shell mishandled valid multiline commands, causing repeated tool timeouts. The evolved agents repaired this behavior. For example, on the Terminal-Bench task count-dataset-tokens, all four multiline commands attempted by the initial agent timed out. The Search 2 capability agent executed eight multiline commands without a timeout and solved the task. This provides a concrete example of how self-editing can improve the reliability of an agent that already has standard tools.

Performance across generations. We also evaluated intermediate checkpoints from searches starting with the original initial agent after search completion, without using their downstream scores to guide search. These checkpoints came from GPT-6.1 Sol searches and were evaluated using GPT-6 Luna.

GenerationMean accuracy (%)
Baseline76.11
176.25
376.81
577.22
776.94
977.36
1078.75

Scores are averaged across both branches and three independent searches. Although scores fluctuate, the displayed checkpoints suggest a modest upward tendency. Standard tools also emerge before the final generation. For example, the capability agent in Search 2 already included exact-text replacement, line-range viewing, and bounded search by generation 5.

These results show that SelfSearch can discover useful improvements without downstream evaluation rewards, including when standard tools are already available.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W2. Weak mechanistic evidence

The paper presents an indirect test of the evolving improver. The fixed-improver ablation only measures final downstream scores, not whether later-generation agents are actually better at editing given the same starting point and budget. Without performance curves across generations, it looks like early generations simply patch obvious tooling gaps while later ones yield diminishing returns.

The performance improvements lack statistical rigor and fair baselines. The reported gains in Table 4 lack error bars or independent runs. Additionally, the evaluation-guided baselines change both the feedback signal and search structure simultaneously, and their comparison are not controlled within the same budget.

Q3. Can you compare early- and late-generation improvers on the exact same starting codebase and budget to directly test whether the agent's ability to self-improve actually gets better?

Evolving improver. Starting from the original initial agent as the target, we compared fixed initial and fixed final improvers with SelfSearch. All searches used the same per-generation editing budget. The fixed-final method instead used final agents from a previous SelfSearch run as frozen improvers. The additional experiments below used GPT-6.1 Sol for search and GPT-6 Luna for downstream evaluation.

MethodCapabilityAdaptive
Fixed initial improver75.8375.83
Fixed final improver77.5076.67
SelfSearch80.0077.50

Values are SWE-bench Verified accuracy (%) on the same 120 tasks. Fixed-initial and SelfSearch results each average three independent searches. The fixed-final result comes from one search.

The fixed-final method scored higher than the fixed-initial mean, but lower than SelfSearch. In SelfSearch, the agent uses and revises its own tools and policies, then carries the revised implementation and the resulting experience into subsequent editing episodes. A fixed improver instead edits a separately evolving target, so experience generated by that target may become less representative of the improver’s own behavior. This comparison evaluates the resulting search outcomes.

Variability across searches.

ConditionCapabilityAdaptive
Initial agent76.11 ± 1.2776.11 ± 1.27
Final agents, three searches80.00 ± 1.6777.50 ± 2.89

Values are mean ± sample standard deviation of accuracy (%) on the same 120 SWE-bench Verified tasks. The initial agent was evaluated three times. Final agents were evaluated once per independent search (three searches).

Both branches improved over the initial agent on average, by 3.89 and 1.39 percentage points, respectively. The capability branch improved in all three searches, while the adaptive branch improved in two of three.

Search cost. Our original experiments compared complete search procedures under matched editing configurations, with total search costs reported explicitly. With GPT, linear and archive search cost \12.35and12.35 and \\7.53, respectively, compared with \6.52forSelfSearch.WithDeepSeek,theycost6.52 for SelfSearch. With DeepSeek, they cost \\8.59 and \7.90,comparedwith7.90, compared with \\4.03 for SelfSearch. These costs include the development evaluations used by the evaluation-guided baselines. SelfSearch achieved competitive performance at lower recorded search cost, although the methods differed in search structure and selection rules.

Experience source under the same search structure. We conducted an additional comparison using the same initial agent, two lineages, ten editing generations, and per-edit execution limits. We replaced inherited self-editing trajectories with task-solving trajectories, patches, and evaluation outcomes. Before each edit, each lineage was evaluated on one task sampled from a ten-task development pool, and both records were shared with both lineages. Development scores did not select or reject candidates.

Experience sourceCapabilityAdaptive
Self-editing experience79.7077.58
Task-solving evaluation experience76.3675.45

Values are mean SWE-bench Verified accuracy (%) across three independent searches per method on the same 110 tasks, excluding the ten development tasks. The SelfSearch scores use the same searches reported above, restricted to these 110 tasks.

Self-editing experience yielded higher mean accuracy by 3.33 and 2.12 percentage points in the capability and adaptive branches, respectively, despite the task-feedback method receiving downstream evaluation outcomes and incurring additional evaluation costs. These results suggest that self-editing trajectories can be more informative for guiding agent improvement: they record attempts to diagnose and revise the agent’s own tools, policies, and execution logic, directly addressing the implementation being improved.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q1. Can you run an ablation that keeps cross-lineage code snapshots and diffs while stripping out the step-by-step editing trajectories? This would isolate the value of the modification process from simple code sharing.

We compared full SelfSearch with a code-only variant that retained cross-lineage code snapshots and diffs but omitted inherited self-editing trajectories. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna.

ConditionCapabilityAdaptive
Full SelfSearch80.0077.50
Code-only sharing75.5676.11

Values are mean SWE-bench Verified accuracy (%) across three independent searches per condition on the same 120 tasks.

Retaining self-editing trajectories improved mean accuracy by 4.44 percentage points for the capability branch and 1.39 for the adaptive branch. Code snapshots and diffs reveal the implementation changes, but do not directly show how those changes affect the agent’s behavior. Self-editing trajectories provide observations of the agent using and revising its tools and policies, including execution failures and checks of the modified behavior. These results support sharing experience of how the implementation behaves alongside the implementation itself.

Reviewer_5MjMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether coding agents can improve their implementations using records of previous self-modification attempts, without downstream tasks or evaluation scores guiding the revisions. SelfSearch maintains two lineages, one for capability expansion and another for adaptive reasoning, which exchange episode records across generations. Each revised implementation becomes the next improver, while model weights and the external execution runtime remain fixed. Agents retain access to tool outcomes and local checks, so "reward-free" refers specifically to the absence of downstream evaluation rewards during search.

The main experiments evaluate the final agents after ten generations using two model configurations on SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench. The mean success rate of the two final agents exceeds the initial agent in all six model–benchmark settings. The authors also report lower search costs than their evaluation-guided baselines, cross-model transfer on SWE-bench Verified, and ablations suggesting that episode records and an evolving improver contribute to final performance.

Critical strengths

The performance of search without downstream evaluation is surprising. The reported searches do not use downstream benchmark tasks or results. I was pleasantly surprised that, for the chosen datasets, the proposed harness is able to match/outperform evaluation driven search strategies.

Critical weaknesses

The authors argue that self-modification provides useful experience because self-modification and downstream coding involve shared activities (code inspection, diagnosis, editing, and verification). The experiments support this explanation with examples of tool failures being repaired and the resulting tools being reused downstream. However, these observations do not establish which properties of the trajectory make it informative about downstream improvement. For example, if an agent pursues an incorrect interpretation of a task, it seems extremely unlikely that reflection on the tool calls or self-consistency checks will yield significant self-correction behavior. Or a reflection attempt that simplifies a tool could introduce a tradeoff that is unfavorable on downstream tasks.

Some potential directions:

  1. Seek out and prove the hypothesis on harder benchmarks. ARC-AGI-3 requires an agent to learn to acquire new tools for solving a task; SWE-Bench Pro V2 is an updated version of SWE-Bench Verified; I recommend looking through the frontier code benchmark set here: https://labs.scale.com/leaderboard. Another
  2. Provide a principled mathematical argument for problems with downstream evaluation; and how they are addressed via the proposed harness. For example, one argument could be that inspecting previous tool calls is providing richer and/or broader information content than downstream evaluations.
  3. Seek out and prove thee hypothesis on a more general set of agentic tasks. For instance, Prompt optimization used to benefit from execution feedback in an evolutionary harness (https://arxiv.org/abs/2507.19457). Does the evolutionary optimizer still benefit from the proposed reward-free method?
  4. What is the variability in the agents the method produces? i.e. Does the proposed algorithm find similar inconsistencies on repeated independent trials? Also, what is variability in the task-solving performance of the final agents? i.e. What is the variance of the final frozen agents perform in multiple independent downstream evaluation runs?
Questions for authors

In weaknesses.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the thoughtful review and questions about how self-editing experience informs downstream improvement.

W1. The authors argue that self-modification provides useful experience because self-modification and downstream coding involve shared activities (code inspection, diagnosis, editing, and verification). The experiments support this explanation with examples of tool failures being repaired and the resulting tools being reused downstream. However, these observations do not establish which properties of the trajectory make it informative about downstream improvement. For example, if an agent pursues an incorrect interpretation of a task, it seems extremely unlikely that reflection on the tool calls or self-consistency checks will yield significant self-correction behavior. Or a reflection attempt that simplifies a tool could introduce a tradeoff that is unfavorable on downstream tasks.

Q2. Provide a principled mathematical argument for problems with downstream evaluation; and how they are addressed via the proposed harness. For example, one argument could be that inspecting previous tool calls is providing richer and/or broader information content than downstream evaluations.

Self-editing can improve downstream performance by repairing tools and execution behavior, even when it does not correct the agent’s interpretation of a task. An agent may understand a task correctly yet fail because its shell mishandles a command or its search tool omits relevant information. Since self-editing and downstream task solving use the same tools and policies, problems encountered during self-editing can reveal obstacles that also affect downstream tasks.

The informative properties of these trajectories are the attempted operations, their observed outputs and failures, and the checks performed after a revision. A downstream score measures task success, but does not by itself explain where execution broke down. Code snapshots show the implementation, but do not directly show how the agent behaves when using it.

For example, an agent encountered failures in multiline Python commands during self-editing. It traced the problem to a shell wrapper that altered commands and repaired the wrapper. The final descendant later used the repaired behavior to diagnose and verify a Django-16661 fix. We also observed changes to behavioral policies. After truncated search results prompted narrower queries that could exclude unread evidence, the agent added search pagination and revised its policy to inspect the remaining results. A subsequent probe showed it following this guidance.

We tested two complementary aspects of this explanation using GPT-6.1 Sol for search and GPT-6 Luna for downstream evaluation.

Code-only sharing. Across three independent searches, retaining code snapshots and diffs while withholding editing trajectories yielded mean accuracy of 75.56% and 76.11% for the capability and adaptive branches, compared with 80.00% and 77.50% for full SelfSearch on the same 120 SWE-bench Verified tasks. These results support retaining observations of how the agent uses and revises its implementation alongside the code itself.

Experience source. We also compared self-editing experience with task-solving evaluation experience. In this experiment, task-solving records described attempts to repair software tasks, whereas self-editing records described attempts to modify the agent’s own tools, policies, and execution logic. The latter therefore provide experience directly concerned with the implementation inherited by subsequent agents.

We retained the same initial agent, two lineages, ten editing generations, and per-edit limits, while replacing inherited self-editing records with task-solving trajectories, patches, and evaluation outcomes. Before each edit, each lineage was evaluated on one task sampled from a ten-task development pool, and both records were shared with both lineages. Development scores did not select or reject candidates.

Experience sourceCapabilityAdaptive
Self-editing79.7077.58
Task-solving evaluation76.3675.45

Values are mean SWE-bench Verified accuracy (%) across three independent searches per method on the same 110 tasks, excluding the ten development tasks. The SelfSearch scores use the same searches reported above, restricted to these 110 tasks.

Self-editing experience does not guarantee correction of mistaken task interpretations or favorable effects from every local revision. Its benefit is that it provides concrete evidence for improving the tools and policies through which the agent acts. The examples illustrate this mechanism, and the comparisons support its usefulness for downstream performance without requiring downstream evaluation rewards during search.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q1. Seek out and prove the hypothesis on harder benchmarks. ARC-AGI-3 requires an agent to learn to acquire new tools for solving a task; SWE-Bench Pro V2 is an updated version of SWE-Bench Verified; I recommend looking through the frontier code benchmark set here: https://labs.scale.com/leaderboard. Another

Q3. Seek out and prove thee hypothesis on a more general set of agentic tasks. For instance, Prompt optimization used to benefit from execution feedback in an evolutionary harness (https://arxiv.org/abs/2507.19457). Does the evolutionary optimizer still benefit from the proposed reward-free method?

We extended the evaluation to DABstep, a data-analysis benchmark, using a sample of 50 tasks from its hard split. These tasks require agents to inspect data, perform computations, and produce answers, extending the evaluation beyond software repair.

We evaluated the initial agent and the final capability and adaptive agents from three independent SelfSearch runs conducted using GPT-5.6 Sol. All downstream evaluations used GPT-6 Luna, without further changes to the agents’ code or policies or DABstep-specific search.

AgentAccuracy (%)
Initial agent32.00
Capability44.00
Adaptive40.00

All agents were evaluated on the same 50 tasks, with answers scored by the official evaluation service. The initial-agent score averages three evaluations of the same implementation. Capability and adaptive scores average the final agents from three independent searches, with one evaluation per agent.

The capability and adaptive agents improved mean accuracy by 12 and 8 percentage points, respectively. These results show that improvements developed through reward-free self-editing can transfer beyond software repair to hard data-analysis tasks without further task-specific adaptation.

Section 3.3 of the revised paper describes the DABstep evaluation and reports accuracy for the initial agent and both SelfSearch lineages on the same 50 hard tasks.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q4. What is the variability in the agents the method produces? i.e. Does the proposed algorithm find similar inconsistencies on repeated independent trials? Also, what is variability in the task-solving performance of the final agents? i.e. What is the variance of the final frozen agents perform in multiple independent downstream evaluation runs?

We examined variability across three independent SelfSearch runs and across three evaluations of the frozen final agents from Search 1. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna.

Variability across searches. We conducted three independent searches from the same initial agent. The searches addressed recurring limitations but produced different additional capabilities. All three produced agents with revised shell execution supporting multiline commands, line-range viewing, and exact-text replacement. Search 1 additionally developed command batching and workflow tools, Search 2 added a helper for comparing checks before and after code changes, and Search 3 developed tools for inspecting background jobs and their outputs.

The resulting agents also varied in downstream performance, as shown below.

ConditionCapabilityAdaptive
Initial agent76.11 ± 1.2776.11 ± 1.27
Final agents, three searches80.00 ± 1.6777.50 ± 2.89

Across independent searches, the capability branch showed higher mean accuracy and lower variability than the adaptive branch. Both branches improved over the initial agent on average, by 3.89 and 1.39 percentage points, respectively.

Variability across repeated evaluations. We evaluated the frozen final agents from Search 1 three times, including the original evaluation and two additional repeats.

AgentCapabilityAdaptive
Initial agent76.11 ± 1.2776.11 ± 1.27
Frozen final agents from Search 179.72 ± 1.7379.44 ± 0.48

Across repeated evaluations of the frozen Search 1 agents, both branches maintained mean accuracy above the initial agent, with standard deviations of 1.73 and 0.48 percentage points, respectively.

Values are mean ± sample standard deviation of SWE-bench Verified accuracy (%) on the same 120 tasks. Across-search results include one evaluation per final agent from each of three independent searches. Initial-agent and frozen-agent results each summarize three evaluations of the same implementation.

Section 4.1 of the revised paper reports mean accuracy across the three independent searches.

In the revised paper, we report the comparison between task-solving and self-editing experience in Section 3.4 and the code-only ablation in Section 4.1. Section 4.3 traces how a tool repaired during self-editing was later used on a downstream task, and Section 6 discusses when self-editing checks can miss errors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

# Review Summary (1/2)

Dear Area Chair,

We thank the reviewers for their feedback. We tested whether agents benefit from inheriting self-editing trajectories alongside code and whether downstream task-solving records offer the same benefit. Across three searches per condition, both SelfSearch lineages achieved higher mean SWE-bench Verified accuracy with self-editing trajectories in each comparison. We also tested agents that started with common tools, evaluated intermediate and frozen agents, and assessed transfer to hard data-analysis tasks.

Strengths noted by reviewers

  • 5MjM. Highlighted SelfSearch’s competitive performance without downstream evaluation during search.
  • tKHk. Found the reward-free search direction interesting.
  • ftfF. Valued the cross-model transfer results and the decision to report both final agents, avoiding checkpoint selection based on downstream scores.

Discussion status

  • ftfF (raised to weak accept). The reviewer found the additional experiments helpful.
  • tKHk (raised to weak reject). The reviewer asked whether the gains mainly reflected repairs to the initial harness. We reported the additional DeepSeek result and documented a case in which the agent changed its search strategy and followed it in a subsequent probe.
  • 5MjM. We responded to the reviewer’s questions about self-editing trajectories, transfer to data analysis, and variability across independent searches.

Paired accuracy scores below are listed in capability/adaptive order.

What does self-editing contribute?

1. Editing trajectories (tKHk)

Concern. Does the editing record add value when agents already inherit code snapshots and diffs?

Response. Across three searches per condition on 120 SWE-bench Verified tasks, sharing code and trajectories yielded 80.00% / 77.50%, versus 75.56% / 76.11% with code alone. Both lineages scored higher when later agents could inspect how earlier edits were made (Section 4.1).

2. Source of experience (5MjM, tKHk)

Concern. Would downstream task-solving records and grading feedback guide revisions just as well?

Response. Both lineages achieved higher mean accuracy with self-editing experience than with task-solving experience that included grading feedback. On the same 110 held-out SWE-bench Verified tasks, the capability lineage averaged 79.70% with self-editing records versus 76.36% with task-solving records. The adaptive lineage averaged 77.58% versus 75.45%. Both conditions used the same search structure and three independent searches (Section 3.4).

3. Changes in agent behavior (5MjM, tKHk, ftfF)

Concern. Does self-editing change how the agent acts beyond repairing faulty tools?

Response. When the search tool cut off its output, the agent tried a narrower query and risked missing matches it had yet to read. It added a way to retrieve the remaining matches and revised its instructions to read them first. A later probe showed it following the revised strategy. We also trace a repaired shell tool to its use in verifying a Django fix (Sections 4.1 and 4.3).

4. Practical lessons for tool design (ftfF)

Concern. What observable difference did a tool revision make, and were any new tools left unused?

Response. On the same input, the original and revised search tools each returned 525 characters. Only the revised version kept the matched keyword visible. We also found that the agent added a helper for running test cases but never invoked it across 120 evaluation tasks (Section 4.3).

AuthorsOpenReview ↗

# Review Summary (2/2)

How robust are the gains?

5. Common tools at initialization (tKHk, ftfF)

Concern. Do improvements persist when the initial agent already has exact-text replacement, line-range viewing, and bounded search?

Response. On Terminal-Bench, the initial agent scored 53.93% in one evaluation. Both GPT lineages exceeded that score in each of three independent searches, averaging 67.42% for capability and 66.29% for adaptive. Neither improved on the 76.67% initial mean on SWE-bench Verified (Section 4.1). From this same starting point, a frozen initial improver reached 75.83% in both branches in one GPT search, compared with 76.67% / 76.11% across three SelfSearch runs (rebuttal). In one DeepSeek search reported in the rebuttal, Verified accuracy rose from 84.17% to 89.17% / 87.50%.

6. Evolving the improver (tKHk, ftfF)

Concern. Does continuing to evolve the improver help, and does a later improver make better individual edits?

Response. Starting from the original initial agent, full SelfSearch averaged 80.00% for capability and 77.50% for adaptive across three searches, versus 75.83% in both lineages with a frozen initial improver. Frozen final improvers reached 77.50% / 76.67% in one search. We evaluated the final agents produced by each search. Whether a later improver makes a better edit given the same code and evidence remains open (Section 4.1).

7. Variability across searches and evaluations (5MjM, tKHk, ftfF)

Concern. Do independent searches discover similar changes, and how much does accuracy vary across searches and repeated evaluations?

Response. Across three independent searches, mean Verified accuracy was 80.00% / 77.50%, with sample standard deviations of 1.67 / 2.89 percentage points. Across three evaluations of the frozen Search 1 agents, the corresponding means were 79.72% / 79.44%, with standard deviations of 1.73 / 0.48 points. All three searches produced support for multiline shell commands, line-range viewing, and exact-text replacement. Other additions varied by run, including command batching in Search 1, a helper for comparing checks before and after edits in Search 2, and background-job inspection in Search 3 (rebuttal).

8. Progress across generations (tKHk, ftfF)

Concern. Are the gains concentrated in the earliest edits?

Response. The initial agent averaged 76.11% across three evaluations. Averaged across three searches and both lineages, checkpoint accuracy was 76.25% at generation 1, 77.22% at generation 5, and 78.75% at generation 10. Accuracy fluctuated across the intermediate checkpoints, with the highest observed mean at generation 10 (rebuttal).

How well do the agents transfer, and at what cost?

9. Transfer to data analysis (5MjM)

Concern. Do the evolved agents improve on hard tasks outside software repair?

Response. On 50 hard DABstep tasks, the initial agent averaged 32% accuracy and final agents from three searches averaged 44% / 40%. Both lineages improved on the same data-analysis tasks under the same evaluation model and official grader (Section 3.3).

10. Matched execution settings (ftfF)

Concern. How much of the high-effort Terminal-Bench result persists when the initial agent receives the same execution settings?

Response. We corrected the original comparison after the provider records identified the served model as DeepSeek V4.1 Flash. In a follow-up evaluation using DeepSeek V4 Flash at maximum reasoning effort for both agents, Terminal-Bench accuracy rose from 74.2% for the initial agent to 78.7% for the evolved capability agent (Section 3.6).

11. Search costs (tKHk, ftfF)

Concern. How do search budgets and candidate selection affect the comparison with evaluation-guided methods?

Response. SelfSearch cost \6.52withGPTand6.52 with GPT and \\4.03 with DeepSeek. Linear and archive search cost \12.35and12.35 and \\7.53 with GPT, and \8.59and8.59 and \\7.90 with DeepSeek. SelfSearch retained the final agent from each lineage. The evaluation-guided baselines selected the candidate with the highest development score and included those evaluations in their search costs. Under these protocols, SelfSearch achieved competitive accuracy at lower observed search cost (Section 3.4 and Appendix C).

Related work

12. Comparison with Gödel Agent (ftfF)

Concern. How does SelfSearch differ from prior agents that can revise their own improvement logic?

Response. We added Gödel Agent to the comparison. It modifies behavior at runtime and uses validation-task performance to guide improvement. SelfSearch passes repository edits and editing records to the next episode and uses those records and local checks to guide revisions (Section 5 and Appendix F).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.