- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SelfSearch improves agent code using records of earlier self-modification episodes, without downstream benchmark feedback during search. Model weights stay fixed. Two agent lineages share records, and each updated agent performs the next editing episode. The authors test the final agents on SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.1. They report higher average success across the two lineages in all six model–benchmark settings, with some cost reductions and cross-model transfer. Ablations and case studies examine the roles of past records, the evolving improver, and the tools it develops.
Critical strengths
-
Using self-editing experience to improve agents is a useful idea. The method avoids repeated downstream evaluation during search. Reporting both final lineages also avoids choosing the main results based on downstream checkpoint scores.
-
The results cover two model settings and three benchmarks. The improved harnesses transfer across model families without further search. Comparing costs on tasks solved by both agents also gives useful evidence of execution efficiency.
-
The case studies show practical changes, such as keeping matching text in search results and linking tool calls to their results across trace pages. The ablations also test two relevant choices: access to past records and whether the improver changes.
Critical weaknesses
-
The initial agent lacks common tools such as exact-text replacement and line-range viewing (Appendix A). It is unclear how much of the gain comes from adding these tools. The fixed-improver ablation provides some support for recursive updating, but only from this limited starting point. The paper does not show whether the benefit remains when both the initial agent and the fixed improver already have these tools.
-
Each model setting has one search with two lineages that share records. These are not independent repeats. Without repeated searches or error bars, it is hard to judge how stable the gains are, especially the 1.7–2.9 percentage-point differences in the fixed-improver ablation.
-
Related work and Table 13 omit Gödel Agent (Yin et al., ACL 2025), which already allows an agent to change both its task policy and its improvement logic. The paper should explain this difference and compare what each method can modify and when those changes take effect.
-
The discovered changes could provide clearer lessons for harness design. Tables 5 and 11 list changes, while Table 12 counts tool use without showing how much each tool helps.
-
The comparisons do not fully separate method gains from budget differences. The evaluation-guided baselines use different search budgets and selection rules (Appendix C.3). The 82.0% Terminal-Bench result also uses higher reasoning effort and larger execution limits, without reporting the initial agent's performance under the same settings. These results do not show the best method at equal cost or SelfSearch's contribution to that high-resource result.
Questions for authors
-
Could the authors start with the common editing and search tools already installed, then compare no search, a fixed improver, and an evolving improver under the same budget? This would directly test the added value of recursive updating.
-
Could the authors repeat the search and report performance across generations? Intermediate agents can be evaluated after search ends, so those results do not guide search. This would show whether gains are stable and continue beyond the early tool additions.
-
Could the authors expand the search-excerpt and trajectory-reader examples to show the original failure, the code change, and the effect on errors, calls, or cost? Examples where a change does not help would also make the design lessons more useful.
7 comments
Official Comment by Authors (2/5)
W2. Each model setting has one search with two lineages that share records. These are not independent repeats. Without repeated searches or error bars, it is hard to judge how stable the gains are, especially the 1.7–2.9 percentage-point differences in the fixed-improver ablation.
Starting from the original initial agent, we conducted three independent searches each for SelfSearch and the fixed initial improver. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna.
| Method | Capability | Adaptive |
|---|---|---|
| Fixed initial improver | 75.83 ± 3.00 | 75.83 ± 1.67 |
| SelfSearch | 80.00 ± 1.67 | 77.50 ± 2.89 |
Values are mean ± sample standard deviation of SWE-bench Verified accuracy (%) on the same 120 tasks. Fixed-initial and SelfSearch results each summarize three independent searches.
Full SelfSearch outperformed the fixed improver by 4.17 percentage points for the capability branch and 1.67 percentage points for the adaptive branch. These results support the benefit of allowing the improver to evolve during search.
W3. Related work and Table 13 omit Gödel Agent (Yin et al., ACL 2025), which already allows an agent to change both its task policy and its improvement logic. The paper should explain this difference and compare what each method can modify and when those changes take effect.
Gödel Agent is closely related to our work, as it can modify both its task-solving policy and improvement logic. We have added it to the related-work discussion and Table 13, and expanded Appendix F to clarify what each method modifies and when changes take effect. Gödel Agent uses runtime monkey patching, while SelfSearch edits repository code that is executed in the next agent episode and carries forward the records of making those edits. A central distinction is the feedback used during search. Gödel Agent’s reported experiments use validation-task performance to guide improvement, whereas SelfSearch uses self-editing records and local checks without downstream evaluation rewards.
Official Comment by Authors (3/5)
W5. The comparisons do not fully separate method gains from budget differences. The evaluation-guided baselines use different search budgets and selection rules (Appendix C.3). The 82.0% Terminal-Bench result also uses higher reasoning effort and larger execution limits, without reporting the initial agent's performance under the same settings. These results do not show the best method at equal cost or SelfSearch's contribution to that high-resource result.
Search costs and feedback comparison. Our original comparison evaluates complete search procedures under matched editing configurations, with search costs reported explicitly. Linear and archive search incurred total search costs of \7.53 with GPT, and \7.90 with DeepSeek, respectively, compared with \4.03 for SelfSearch. These costs include the development evaluations used to guide the baselines, while SelfSearch requires no downstream evaluations during search. SelfSearch therefore achieved competitive performance at lower recorded search cost, despite differences in search structure and selection rules.
To examine the role of feedback while retaining the search structure, we conducted an additional comparison using the same initial agent, two lineages, ten editing generations, and per-edit execution limits. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna. This comparison contrasts experience from solving downstream tasks with experience from modifying the agent’s own tools, policies, and execution logic. We replaced inherited self-editing trajectories with task-solving trajectories, submitted patches, and evaluation outcomes. Before each edit, each lineage was evaluated on one task sampled from a ten-task development pool, and both records were shared with both lineages. Development scores did not select or reject candidates.
| Experience source | Capability | Adaptive |
|---|---|---|
| Self-editing experience | 79.70 | 77.58 |
| Task-solving evaluation experience | 76.36 | 75.45 |
Values are mean SWE-bench Verified accuracy (%) across three independent searches for each method, evaluated on the same 110 tasks excluding the ten development tasks. The SelfSearch scores use the same three searches reported above, restricted to these 110 tasks.
Self-editing experience yielded higher mean accuracy by 3.33 and 2.12 percentage points for the capability and adaptive branches, respectively. Both methods had the same editing opportunities, while the task-feedback method incurred additional development-evaluation costs. These results suggest that self-editing trajectories can provide more useful guidance for agent improvement under this search structure.
Correction to the Terminal-Bench comparison. For the original high-resource Terminal-Bench comparison, we requested DeepSeek V4 Flash through OpenRouter, but the inspected OpenRouter response records identified the served model as DeepSeek V4.1 Flash. We have corrected the manuscript and withdrawn the original claim of a model-matched comparison with the published Codex result.
In a follow-up evaluation using DeepSeek V4 Flash at max reasoning effort, SelfSearch solved 70 of 89 tasks (78.7%), compared with 66 (74.2%) for the initial agent. This score would rank second among the scores reported in the nine-harness comparison, behind Codex.
Official Comment by Authors (1/5)
We thank the reviewer for the constructive feedback and suggestions for additional controls, quantitative analysis, and clearer design lessons.
We started from an initial agent already equipped with exact-text replacement, line-range viewing, and bounded search. We compared no search, a fixed initial improver, and SelfSearch. The fixed-improver method used the same initial agent as both the frozen editor and the starting target, with the same per-generation editing budget as SelfSearch. Searches used GPT-6.1 Sol, and downstream evaluations used GPT-6 Luna.
Comparison with a fixed improver.
Values are SWE-bench Verified accuracy (%) on the same 120 tasks. The initial-agent score averages three evaluations. Fixed-initial results come from one search, while SelfSearch results average three independent searches.
With standard tools already available, SelfSearch scored slightly higher than the fixed improver on SWE-bench Verified, while remaining close to the initial agent’s performance.
Evaluation on Terminal-Bench. We also evaluated the final SelfSearch agents on Terminal-Bench to examine whether improvements from this starting point extend to a broader set of terminal tasks.
All searches started from an initial agent equipped with standard editing and search tools. All agents were evaluated on the same 89 Terminal-Bench 2.1 tasks. Capability and adaptive scores average the final agents from three independent searches. The initial-agent score comes from one evaluation.
Both branches exceeded the initial agent in all three searches, with mean gains of 13.48 and 12.36 percentage points. These results show that improvements remain possible when standard editing and search tools are already present, although the gains differ across benchmarks.
The improvements included repairs to execution behavior. The initial agent’s shell mishandled valid multiline commands, causing repeated tool timeouts. The evolved agents repaired this behavior. For example, on the Terminal-Bench task
count-dataset-tokens, all four multiline commands attempted by the initial agent timed out. The Search 2 capability agent executed eight multiline commands without a timeout and solved the task. This illustrates how self-editing can improve the reliability of an agent that already has standard tools.