acceptodds
Under review as a conference paper at ICLR 2027

Branch-reset: Escaping Idea Basins in Autoresearch Agents

Abstract

Autoresearch agents tackle open-ended problems by repeatedly proposing candidate solutions, evaluating them, and using feedback to guide subsequent experiments. We show that independent runs of the same agent on the same task often plateau at substantially different scores, with gaps that persist even after considerable additional compute. Embedding their candidate artifacts by functional similarity provides further evidence that trajectories remain in localized regions of the solution space, which we call idea basins. To help agents escape these basins, we study a simple periodic intervention, branch-reset. It forks the agent into parallel trajectories that each inherit a copy of the accumulated workspace but begin with fresh chat contexts. After running each trajectory for a fixed horizon, the agent continues from the highest-scoring one. Across long-horizon research and engineering tasks, with individual agent runs lasting up to several days, branch-reset outperformed the single-run and best-of- baselines by a relative improvement of % and %, respectively, on the min–max normalized average score under an equal compute budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Test-Time Compute, Open-Ended Problems, Long-Horizon Agents, Auto Research Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NyDgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Branch-Reset, a harness intervention on a long horizon autoresearch agents. A run opens with K=4 agents starting from an empty workspace, and when 33% and 66% of the budget has been spent, the best-scoring agent is forked into K=4 new agents that inherit its workspace but start with an empty chat context. Each fork runs for about 0.06B credits, and the run continues from the highest-scoring fork. The authors reported consistent improvement over single-run and best-of-4 baselines on 13 tasks. An ablation on 11 tasks attributes the gain to both branching and context reset.

Critical strengths
  1. The problem is important and timely. The paper documents clearly that autoresearch runs plateau at persistently different levels. This observation is useful to the community.
  2. The empirical study is large and costly. It covers 13 open-ended tasks with individual runs of up to 107 hours and about 7M credits in total.
  3. The authors took great effort in defining and visualizing the “idea basins”, and I appreciate this effort.
  4. The appendix discloses all agent prompts and reports a negative result: plateau-triggered schedules rarely fired. This transparency is appreciated.
Critical weaknesses

W1. "Equal compute" is matched only on LLM tokens, which may systematically favor the reset conditions. Credits are a function of input and output tokens. An agent with a fresh context pays far fewer input tokens per step than one carrying a long history, even a compacted one. For the same credits, it can therefore take more actions and make more graded submissions. The paper reports no step counts, submission counts, or tokens per step for any condition.


W2. The improvement is concentrated in a few tasks, and the empirical improvement looks fragile.

(a) The variance of the results in Table 1 is high. In the 13 tasks, most tasks do not have significant improvement over baselines. On some tasks Branch-Reset is even worse than Single Run. The authors did not try to explain why this happened.

(b) The head line “66%” or “44%” improvement is computed on min-max normalized scores and does not make sense. It is like 20℃ is not twice as hot as 10℃

(c) The ablation is similarly fragile. The claim that branching and resetting are both needed is clearly supported on only four of the eleven tasks: mainly “AHC014” and “order-add”. These largest gaps come from the highest-variance cells (e.g., SD 29.35 on AHC014 with n=5).


W3. The basin-escape mechanism is not sufficiently supported.

(a) The only quantitative analysis of a jump covers a single round on polyomino (Fig. 3). Appendix G notes that many rounds produce no basin jump yet still improve the score, which suggests the gain does not require basin escape.

(b) The paper does not report, across all mid-run rounds, how often the selected branch lands in a new basin. Nor does it compare gains between rounds with and without a jump. A more detailed behaviour analysis is needed here.


W4. Methodological novelty is limited, and the baselines do not represent the closest prior work.

(a) Branching with selection is not new. As the related-work section itself acknowledges, it is go-with-the-winners, or the exploit step of Population Based Training.

(b) Context reset with a persistent workspace is not new either. Works like FunSearch and AlphaEvolve, both cited in the paper, build every LLM query from a fresh prompt that contains only the top-scoring programs.

(c) The only population baseline, best-of-4, is weak by construction. B is set where single runs plateau, so each B/4 run is cut off before it plateaus. Stronger baselines are needed to isolate the contribution, e.g., successive halving, an early best-of-4 followed by a single run, and periodic restore-to-best with a fresh context but no branching. The authors should also discuss recent work on population-based autoresearch (e.g., GEAR) and on context contamination when retrying.

Questions for authors

See the above weakness section. I will reconsider the ratings based on the authors' responses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer NyDg (Part 1/2)

Response to Reviewer NyDg (Part 1/2)

We thank the reviewer for the thorough feedback and for noting the problem's importance, the scale of the study, and the value of the idea-basin visualization. Due to character limits, we split our response into two comments. In this first comment (Part 1), we address W1-W3. Please see Part 2 for W4 and concluding remarks.

W1: Equal Compute, Tokens, and Submissions

We agree with the reviewer that compute matching should be carefully controlled. We added three controls. First, we evaluated Qwen3.6-27B under a processed-token budget, W = input - cached input + output:

Tasksingle-runbest-of-4succ-halvingbranch-reset
polyomino80.079.779.877.1
job-shop6.56.97.57.5
AHC0146.48.816.17.4
p15067.458.066.774.1
p17162.651.851.763.9
IQM0.35 [0.23,0.49]0.29 [0.20,0.39]0.42 [0.34,0.53]0.48 [0.37,0.59]

Second, in the main credit-matched runs, branch-reset uses a median 1.18x output tokens and 1.46x graded submissions. Truncated at the median single-run total on each task, it still beats single-run on 5/6 tasks at equal output tokens and 7/11 tasks at equal submissions.

Third, extra submissions do not fully explain the gain: on sphere-pack, LWE-rec, and LLM-SQL, where branch-reset makes the most extra submissions, it gains under 3% or loses; on Borden-inv, it makes fewer submissions while gaining about 33%. The processed-token budget addresses cached context: cached history is not charged, so long context does not make single-run steps more expensive, while reset branches still pay to re-read workspace files. We added the processed-token evaluation in Section 5.4 and token/submission controls in Appendix H.

W2: Variance, Aggregate Metrics, and Ablation

We value the reviewer's comment regarding variance and aggregate metrics. We added directional tests, IQM, raw-score relative improvements, and a clearer explanation of where gains are largest. Branch-reset has a higher mean than single-run on 12/13 tasks and than best-of-4 on all 13. On the eleven GPT-5.6-terra tasks, 49/55 branch-reset runs exceed the task's median single-run score. It lowers raw standard deviation on 10/13 tasks; it rises on AHC014 and order-add because jumps raise the upper tail, and on LWE-rec.

The gain size tracks single-run spread. AHC014, tinyCIFAR, Borden-inv, and order-add have the widest spread and largest gains. Low-spread tasks such as relay-lay and LLM-SQL gain under 1%. LWE-rec is the only task where branch-reset trails single-run; its runs end in a narrow band and resets spend budget re-orienting.

We replaced normalized-ratio headlines with IQM and raw-score relative gains. Table 1 now reports IQM with 95% stratified-bootstrap intervals: 0.86 [0.83,0.92] for branch-reset, 0.63 [0.44,0.65] for best-of-4, and 0.51 [0.39,0.58] for single-run. On native raw scores, branch-reset improves over single-run and best-of-4 by averages of 26.7% and 24.2% and medians of 6.9% and 4.2%.

For ablations, branch-reset has the highest mean on 8/11 tasks and beats reset-only and branch-only on 9/11. Aggregate IQM rises from 0.44 [0.31,0.50] single-run to 0.57 [0.44,0.67] reset-only, 0.62 [0.52,0.70] branch-only, and 0.80 [0.75,0.87] branch-reset. The tinyCIFAR checkpoint test further isolates context reset: 15/20 reset branches escape score plateaus versus 0/20 branches that keep history. We added these metrics in Table 1 and Appendix A.

W3: Basin-Escape Mechanism and Round-Level Behavior

We agree that more behavior analysis strengthens the mechanism claim. We added both aggregate jump analysis and a controlled checkpoint test. Score gains do not require an idea-space basin jump in every round: branch-reset helps through fresh-context refinement near the inherited workspace and occasional large jumps. The controlled tinyCIFAR test measures score-plateau escape, not necessarily idea-space movement: clearing context breaks stagnation with 15/20 reset branches escaping versus 0/20 branches that retain history.

We added a jump analysis across the five mapped tasks: 90 valid mapped reset branches over 24 rounds. A large move is when a reset branch's first artifact is farther from its branch point than 99% of single-run steps after the first quarter of the budget; reset branches make such moves 9/90 times, and at least one branch jumps in 9/24 rounds. Jump sizes range from 12x to 153x the median single-run step.

A large-move branch is selected in 2/24 total mapped rounds, i.e. 2/9 jump rounds. This is expected because a branch that changes approach often starts from an untuned implementation. In both selected-jump rounds, however, it gives the largest single-round gain on its task. We added the controlled context-reset test in Appendix J and the mapped-jump analysis in Appendix O of the revised version.

AuthorsOpenReview ↗

Continued Response to Reviewer NyDg (Part 2/2)

Continuing from our previous response, we address the remaining weakness below.

W4: Novelty, Baselines, and Related Work

We agree with the reviewer that novelty and baseline positioning should be stated precisely. To address this point, we clarified the novelty claim, added stronger baselines, and expanded related work. We do not claim branching or context reset alone as new. The novelty is diagnosing context-induced stagnation in stateful long-horizon autoresearch agents and showing that workspace branching plus periodic context reset mitigates it. FunSearch and AlphaEvolve issue stateless, single-turn calls from stored programs, so no multi-turn agent history accumulates. Our agents run for hours to days in persistent workspaces with accumulated tool calls, logs, and reasoning, where keeping history is the default.

Both components matter, and the added baselines sharpen where the method helps:

Taskbest-of-2best-of-4best-of-8branch-reset
polyomino91.590.792.594.4
job-shop9.929.919.8510.33
AHC01427.020.721.457.5
p150122.6123.5124.5129.7
p17193.388.993.092.8
sphere-pack43.643.843.744.9
relay-lay72.5972.7472.7272.81
LWE-rec149.6141.1116.2144.6
LLM-SQL26.9326.9327.0027.04
order-add25.526.228.249.5
Borden-inv70.076.576.585.9
kernel126.2122.5119.3135.2

Branch-reset is higher than all three fixed-budget best-of-N settings on 10/12 tasks, though larger N also makes each independent run shallower. In the component ablation, single-run/reset-only/branch-only/branch-reset reach IQM 0.44 [0.31,0.50] / 0.57 [0.44,0.67] / 0.62 [0.52,0.70] / 0.80 [0.75,0.87]. In the Qwen suite, succ-halving reaches 0.42 IQM versus 0.48 for branch-reset; this is a smaller, mixed result with overlapping intervals, not a uniform win.

Reset-only tests fresh context without branching. For Qwen models, succ-halving is above single-run and best-of-4 in IQM and beats branch-reset on polyomino and AHC014, while branch-reset leads on p150 and p171 and the two tie on job-shop.

We expanded related work to discuss go-with-the-winners, Population Based Training, FunSearch, AlphaEvolve, GEAR, SwarmResearch, Heuresis, MLEvolve, and context contamination when retrying across turns. Branch-reset differs by keeping one lineage, preserving workspace state, and discarding conversational history at selected forks. We added the positioning in Related Work, the main baseline results in Table 1, the open-source baseline in Section 5.4, and fixed-budget best-of-N scaling in Appendix G.

Concluding Remarks

We thank the reviewer again for the detailed feedback and constructive suggestions. We added equal-output-token and equal-submission controls, processed-token open-weight evaluation, improved aggregate metrics, ablation confidence intervals, controlled context-reset tests, all-round basin-jump quantification, per-round analysis, stronger baselines, and expanded related work. We hope that our rebuttal was sufficient in addressing all the important questions raised by the reviewer and allows them to endorse our paper even more strongly with a potential score upgrade if they deem it appropriate.

AuthorsOpenReview ↗

Follow-Up on Our Response to Reviewer NyDg

Dear Reviewer NyDg,

We would sincerely appreciate hearing whether our response and new analyses addressed the questions you raised as the discussion deadline approaches. In brief, we added processed-token, equal-output-token, and equal-submission controls; reported IQM and ablation confidence intervals; quantified score-plateau recovery and mapped idea-space jumps; added fixed-budget best-of-N, reset-only, and Qwen successive-halving comparisons; and expanded our novelty and related-work discussion. Thank you again for your detailed feedback and for recognizing the importance, scale, and transparency of our work.

Reviewer_vrhdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates stagnation in long-horizon autonomous research agents, showing that independent runs can converge to localized regions of the solution space, which they call “idea basins.” The authors propose branch-reset, which periodically forks execution, resets conversational context while preserving the workspace, and continues from the highest-scoring branch. Across 13 tasks spanning (Frontier-CS, EdgeBench), the method achieves the highest mean score on 12 tasks. Under matched compute budgets, it improves the aggregate min–max normalized score by 66.0% over single-run execution and 44.4% over a best-of-four baseline.

Critical strengths
  • The paper tackles an important problem in current LLM-based research on how we can increase diversity and performance with agents open-ended research.
  • The method is simple and practical. It needs no training or extra models, just a small change to the agent harness that anyone running long agent loops could adopt.
  • The functional-similarity mapping offers a useful diagnostic tool for studying how agents explore solution spaces beyond tracking scores alone.
  • The core observation, that independent runs of the same agent plateau at very different scores even with lots of extra compute, is useful and well documented across tasks.
Critical weaknesses
  • The experimental setting is limited. It only addresses 13 tasks, selected from different sources. The authors do not mention why they use a different model for 2 tasks, and it is not explicitly mentioned in the results table. Thus, generalizability remains uncertain with a single agent harness and a limited set of models, and four or five repetitions per task.
  • Regarding the method, it relies on context resetting and stochastic resampling to induce diversity. The basin analysis illustrates successful escapes, but an aggregate assessment of escape frequency and compute spent on redundant branches would better establish when branch-reset provides useful exploration and when its additional sampling is inefficient. One could argue that the evidence that basins cause plateaus is mostly a few hand-picked visualizations, and the MAX-3-SAT inverted-U is what you’d expect from any bounded score.
  • The gains can’t be cleanly credited to the context reset, because two of the four branches are also explicitly told to try a fundamentally different approach, and there’s no ablation without that instruction. The evidence is thin, and the headline “66% improvement” is a ratio of min–max normalized scores, which isn’t a meaningful effect size.
Questions for authors
  • Why did you use GPT-5.5 instead of GPT-5.6-terra for kernel and tinyCIFAR, and do the results hold if all tasks use the same model? Have you tried branch-reset with a different agent harness or an open-weight model?
  • Across all branch-reset rounds, how often does the selected branch actually move to a new basin, and how does that compare to single runs over the same compute? This leads to another question on how often does the selected branch actually move to a new basin, and how does that compare to single runs over the same compute?
  • How much does each branching round contribute to the final gain, and does performance keep improving with more rounds or level off after the first few?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vrhd (Part 1/2)

Response to Reviewer vrhd (Part 1/2)

We thank the reviewer for the feedback and for noting the importance of the problem and the usefulness of the idea-basin observation. We address each weakness (W) and question (Q) in order. Due to limits, we split our response into two comments. Here (Part 1), we address W1-W3 and Q1. Please see Part 2 for Q2-Q3 and concluding remarks.

W1: Experimental Setting and Open-Weight Model Evaluation

We thank the reviewer for asking for clarification on model choice, task count, and harness generalization. We clarified the model used for every task: kernel and tinyCIFAR use GPT-5.5, and the remaining eleven tasks use GPT-5.6-terra. Kernel and tinyCIFAR were evaluated first, when GPT-5.5 was the strongest available model. Within every task, all methods use the same model. branch-reset has a higher mean than single-run on 10/11 tasks evaluated with GPT-5.6-terra and best-of-4 on all 11 ones. We added this clarification to Table 1 and its caption.

To add more evidence, we evaluated Qwen3.6-27B in OpenAI Codex CLI, five seeds per condition, matched at B=5M processed tokens:

Tasksingle-runbest-of-4succ-halvingbranch-reset
polyomino80.079.779.877.1
job-shop6.56.97.57.5
AHC0146.48.816.17.4
p15067.458.066.774.1
p17162.651.851.763.9
IQM0.35 [0.23,0.49]0.29 [0.20,0.39]0.42 [0.34,0.53]0.48 [0.37,0.59]

Branch-reset has the highest IQM and improves over single-run and best-of-4 by 7.7% and 8.4% in mean raw score, but the pattern is mixed: it trails single-run on polyomino and succ-halving on AHC014, and IQM intervals overlap. We added this in Section 5.4 and its table.

The task count is limited by cost: runs take hours to days and budgets range from 6k to 100k credits. The set spans algorithmic optimization, cryptanalysis, scientific inference, and systems engineering, and was not selected by outcome.

W2: Quantification of Idea Basin Jumps

We value the reviewer's comment regarding aggregate evidence beyond selected visualizations. We added an all-round jump analysis over 90 reset branches with valid idea-space embeddings across 24 rounds. A large move is when a reset branch's first artifact lands farther from its branch point than 99% of single-run steps after the first quarter of the budget; reset branches make such moves 9/90 times (10%), with at least one jump in 9/24 rounds. Jump distances are 12x to 153x the median single-run step, and the large-move branch is selected in 2/24 mapped rounds. We added this in Appendix O and point to it from the Analysis section.

To further validate the mechanism, we added a controlled tinyCIFAR checkpoint test: context reset escapes 15/20 score plateaus, compared with 0/20 branches that keep history, with workspace, budget, and prompt held fixed. This "escape" is score-plateau recovery, distinct from idea-space basin movement. We added this in Appendix J.

We also added discarded-branch accounting in Appendix H. The overhead is included in the equal-budget comparison: it helps on high-spread tasks such as AHC014, order-add, and Borden-inv, but is inefficient on low-spread tasks such as relay-lay and LLM-SQL. We clarified the MAX-3-SAT analysis in the revised Analysis section: the high-variance regime shifts with model capability.

W3: Context Reset, Diversity Prompt, and Aggregate Metrics

We agree that the diversity instruction should be separated from context reset. Branch-only uses the same branching schedule and diversity prompt while retaining history, reaching 0.62 IQM versus 0.80 for branch-reset. Across 178 diversity-prompted branch choices, an instructed branch is selected 81 times (46%), consistent with chance. In the tinyCIFAR checkpoint test, neither arm receives a diversity instruction; reset branches escape 15/20 score plateaus versus 0/20 with history. We added this in Appendix B and the ablation section.

To address the aggregate-metric point, we replaced normalized-ratio headlines with IQM and raw-score relative gains. Table 1 now reports IQM with 95% stratified-bootstrap intervals: 0.86 [0.83,0.92] for branch-reset, 0.63 [0.44,0.65] for best-of-4, and 0.51 [0.39,0.58] for single-run. We added IQM to Table 1 and raw-score gains to Appendix F (average 26.7%/24.2%, median 6.9%/4.2% over single-run/best-of-4).

Q1: Model Choice and Open-Weight Evaluation

As described in W1, we clarified model usage in Table 1 and added the Qwen3.6-27B evaluation. These additions provide evidence from a second harness/model, with smaller and mixed gains rather than a uniform win. We added them in Table 1 and Section 5.4 (Ablation: Open-Source Model) of the revised manuscript.

AuthorsOpenReview ↗

Continued Response to Reviewer vrhd (Part 2/2)

Continuing from our previous response, we address the remaining questions below.

Q2: How Often Branches Move to a New Basin

We value the reviewer's question regarding how often branches move to a new basin. We added the all-round mapped-jump analysis to answer it directly: large moves occur in 9/90 mapped branches, with at least one jump in 9/24 rounds. The large-move branch is selected in 2/24 total mapped rounds, i.e. 2/9 jump rounds; in both selected-jump rounds, it gives the largest mapped round gain on its task. We added this analysis in Appendix O of the revised version and point to it from the Analysis section.

Q3: Contribution of Each Branching Round

Across 101 completed mid-run branch-reset rounds on the eleven GPT-5.6-terra benchmark tasks, 98/101 rounds raise the run's best score. This count is different from the 178 diversity-prompted branch choices in Appendix B. The branching rounds account for more than half of post-opening gain on 7/11 tasks, and gains taper: the first mid-run reset contributes more than the second on 10/11 tasks. In our cadence study, both one reset at 50% and two resets at 33% and 66% beat single-run on all six tested tasks. We added the per-round contribution analysis in Appendix I and the cadence study in Appendix E of the revised version.

Concluding Remarks

We are grateful for the reviewer's feedback, which helped us make the empirical story clearer. We added model disclosure, Qwen3.6-27B Codex CLI results, aggregate jump analysis, the controlled checkpoint test, discarded-branch accounting, improved aggregate metrics, and per-round contribution analysis. We hope these additions address the points raised by the reviewer, and we are happy to answer further questions. If satisfied, we would be encouraged if the reviewer considered a fresh evaluation and score upgrade if merited.

AuthorsOpenReview ↗

Follow-Up on Our Response to Reviewer vrhd

Dear Reviewer vrhd,

We would be very grateful to hear whether our response addressed the questions you raised as the discussion deadline approaches. In brief, we clarified the model choices across tasks, evaluated Qwen3.6-27B in a second harness, quantified mapped idea-space jumps and discarded-branch overhead, isolated context reset from diversity prompting, added IQM and raw-score aggregate metrics, and analyzed the contribution of each branching round. Thank you again for recognizing the importance and usefulness of our work, and for your thoughtful feedback.

Reviewer_9MQkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a key limitation in today's methods for agent autoresearch, which is that agents fall into plateaus in the diversity of their ideas and get stuck in a limited solution space called idea basins. To escape this, the paper proposes branch reset, a method that generates ideas in parallel branches and keeps ones that score highly along with a reset mechanism that removes previous context from the branch while keeping the same workspace. As a result, the agent picks up from previous progress while not relying heavily on previous context. On long horizon tasks, branch reset outperforms the best-of-N baseline.

Critical strengths

Interesting and empirical discovery of idea basins. This paper proposes a concrete and quantitative way to detect and analyze idea basins, which is motivated by a intuitive finding that agents get stuck in a restricted solution space. This approach offers a clean way to compare methods in future research, which is a strong contribution.

The ablation experiments are clear and provide strong evidence that branching and reset are both necessary components. The analysis also clearly shows that context reset lets the agent escape idea basins.

The analysis on model variance by using Max-SAT as a motivating example was insightful. Again, the paper takes an intuitive idea that agent capability interacts with model variance (either too easy or too hard results in low variance) and concretely formalizes the idea.

Critical weaknesses

The major concern that I have is how branch reset scales with respect to compute compared to a simple method like best-of-N. In Figure 3 left, the trajectory shows a big jump at this context reset step. However, the artifacts go from a close point to basin 2 to a point near basin 3. This suggests that the agent does not actually explore novel areas. It is true that the final solution discovered is outside the basin 3 area, but the very first artifacts are also outside basin 2, which suggests that this may be due to randomness. I wonder if this method is still limited by the inherent diversity of the model and as we scale n in best-of-N, this will eventually capture all the diversity and branch-reset will underperform in this setting.

Novelty: branching is a common technique already used for agentic interactions (beam search [1], tree of thoughts [2], Language Agent Tree Search [3]). Similarly, context reset is also a studied technique [4].

The main issue dealt by the paper is low diversity of solutions. As such, it would be beneficial to the paper if a more diverse set of models and agent harnesses were used.

[1] Xie, Yuxi, et al. "Self-evaluation guided beam search for reasoning." Advances in Neural Information Processing Systems 36 (2023): 41618-41650. [2] Yao, Shunyu, et al. "Tree of thoughts: Deliberate problem solving with large language models." Advances in neural information processing systems 36 (2023): 11809-11822. [3] Zhou, Andy, et al. "Language agent tree search unifies reasoning acting and planning in language models." arXiv preprint arXiv:2310.04406 (2023). [4] Laban, Philippe, et al. "Llms get lost in multi-turn conversation." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

How does this method compare to others that explicitly encourage diversity from the model (e.g. via prompting, entropy-based, etc)?

It was interesting to see the analysis on the Max-SAT problem. Do you have any insights to where the actual tasks lie in this spectrum? Which tasks are below agents capacity vs above vs sit at the sweet spot? Does this translate to how effective branch reset was?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9MQk (Part 1/2)

Response to Reviewer 9MQk (Part 1/2)

We thank the reviewer for the thoughtful and encouraging feedback, especially the recognition that our empirical discovery of idea basins and our quantitative way to analyze them are strong contributions. We address each weakness (W) and question (Q) in order. Due to character limits, we split our response into two comments. In this first comment (Part 1), we address W1-W2. Please see Part 2 for W3 and Q1-Q2.

W1: Scaling Compared to Best-of-N and Basin Exploration

We agree with the reviewer that scaling against best-of-N is central to evaluating branch-reset. To address this point, we added a fixed-budget best-of-N study across N=2,4,8 on 12 tasks:

Taskbest-of-2best-of-4best-of-8branch-reset
polyomino91.590.792.594.4
job-shop9.929.919.8510.33
AHC01427.020.721.457.5
p150122.6123.5124.5129.7
p17193.388.993.092.8
sphere-pack43.643.843.744.9
relay-lay72.5972.7472.7272.81
LWE-rec149.6141.1116.2144.6
LLM-SQL26.9326.9327.0027.04
order-add25.526.228.249.5
Borden-inv70.076.576.585.9
kernel126.2122.5119.3135.2

Branch-reset is higher than best-of-2 on 10/12 tasks, best-of-4 on 12/12, best-of-8 on 11/12, and all three N values on 10/12. The exceptions are LWE-rec at N=2 and p171 at N=2 and N=8. This fixed-budget table makes larger N shallower, so we also added the stricter full-budget single-run comparison: branch-reset's mean exceeds the maximum full-budget single-run score on 5/11 benchmark tasks, and its maximum exceeds single-run's maximum on 10/13 tasks.

To add aggregate evidence beyond Figure 3, we added two checks. Across all mapped mid-run rounds, we measure every reset branch with a valid idea-space embedding: 90 mapped branches across 24 rounds. A large move is when a reset branch's first artifact lands farther from its branch point than 99% of single-run steps after the first quarter of the budget. Reset branches make such moves 9/90 times (10%), a 10x increase over the single-run tail rate, with at least one large move in 9/24 rounds. Jump distances range from 12x to 153x the median single-run step. The large-move branch is selected in 2/24 mapped rounds; in both selected-jump rounds, it produced the largest mapped single-round gain on its task.

Finally, we added a controlled tinyCIFAR checkpoint test. Here "escape" means leaving a score plateau, not necessarily jumping to a new idea-space basin. From 8 plateaued checkpoints, we branch with and without clearing chat context, keeping the workspace and budget identical and giving neither arm a diversity directive. Fifteen of 20 context-reset branches escape the plateau, compared with 0 of 20 branches that keep chat history. Thus the plateau is not simply a workspace or model ceiling: clearing accumulated context changes the trajectory. We added fixed-budget best-of-N scaling in Appendix G, the controlled context-reset test in Appendix J, and mapped-jump analysis in Appendix O of the revised version.

W2: Novelty Relative to Tree Search and Context Reset

We thank the reviewer for pointing us to the relevant literature. We do not claim branching or context reset in isolation as new. Our contribution is diagnosing context-induced stagnation in stateful long-horizon autoresearch agents and showing that session-level workspace branching plus context reset mitigates it. Step-level search methods such as self-evaluation guided beam search, Tree of Thoughts, and LATS branch over thoughts or short actions while retaining ancestor context. Laban et al. study multi-turn context degradation, but not persistent code workspaces or population selection. Branch-reset operates at the multi-hour session level: it preserves the evolved workspace, discards accumulated conversation, and selects by the task's own grader. The ablation IQM scores show both pieces matter: single-run 0.44 [0.31,0.50], reset-only 0.57 [0.44,0.67], branch-only 0.62 [0.52,0.70], and branch-reset 0.80 [0.75,0.87]. We updated the revised Related Work and the ablation section to make this distinction explicit.

AuthorsOpenReview ↗

Continued Response to Reviewer 9MQk (Part 2/2)

Continuing from our previous response, we address W3 and the remaining questions below.

W3: Additional Model and Harness

To add evidence beyond the original model/harness, we added an open-weight Qwen3.6-27B Codex CLI evaluation on five algorithmic tasks, five seeds per method, matched at B=5M processed tokens:

Tasksingle-runbest-of-4succ-halvingbranch-reset
polyomino80.079.779.877.1
job-shop6.56.97.57.5
AHC0146.48.816.17.4
p15067.458.066.774.1
p17162.651.851.763.9
IQM0.35 [0.23,0.49]0.29 [0.20,0.39]0.42 [0.34,0.53]0.48 [0.37,0.59]

Branch-reset has the highest IQM and leads on p150 and p171, but the task-level pattern is mixed: it ties succ-halving on job-shop, trails single-run on polyomino, and trails succ-halving on AHC014; the IQM intervals also overlap. We present this as additional evidence in a second harness/model, not as a standalone proof of model-family invariance. We added this in Section 5.4 (Ablation: Open-Source Model) of the revised version.

Q1: Explicit Diversity Methods

We value the reviewer's question about explicit diversity methods. To address it, we added a diversity-prompt analysis. Branch-only uses the same branching schedule and diversity prompt while keeping chat history, reaching 0.62 IQM versus 0.80 for branch-reset. Across 178 diversity-prompted branch choices, a directed branch is selected 81 times (46%), consistent with chance relative to 50%. Most directly, the tinyCIFAR checkpoint test gives neither arm a diversity prompt, yet context reset yields 15/20 score-plateau escapes versus 0/20 without reset. Thus the gain is not explained by the diversity prompt alone. We added these results in Appendix B of the revised version and the ablation discussion.

Q2: Where Tasks Sit on the MAX-3-SAT Spectrum

Open-ended tasks lack exact reference solvers, but single-run score dispersion and progress from B/8 to B map them into analogous regimes. Fast-saturating tasks such as relay-lay have little spread and gains below 1%. Barrier-limited tasks such as LLM-SQL and LWE-rec also show little useful divergence. The frontier regime contains tasks where runs settle into basins of very different quality; AHC014, tinyCIFAR, Borden-inv, and order-add have the widest spread and largest gains. This matches the MAX-3-SAT picture: branch-reset helps most in the high-variance frontier regime. We added this regime analysis in the revised Analysis section, with supporting diagnostics in the appendix.

Concluding Remarks

We are grateful for the reviewer's actionable feedback. In the revision, we added best-of-N scaling, aggregate basin-escape analyses, the controlled checkpoint test, token-matched Qwen3.6-27B results, expanded related work, and prompting and per-task regime analyses. We hope these additions address the reviewer's questions, and we are happy to discuss further. If the reviewer is satisfied, we would be thrilled if they considered strengthening their evaluation further if merited.

AuthorsOpenReview ↗

Follow-Up on Our Response to Reviewer 9MQk

Dear Reviewer 9MQk,

We would greatly value hearing whether our response resolved the points raised as the discussion deadline approaches. In brief, we added a fixed-budget best-of-N scaling analysis, quantified mapped idea-space jumps, conducted a controlled context-reset test, clarified our novelty relative to prior branching methods, and evaluated Qwen3.6-27B in a second harness. Thank you for recognizing our empirical discovery and quantitative analysis of idea basins as strong contributions, and for your thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response Summary

We thank the reviewers (9MQk, vrhd, and NyDg) for their valuable feedback and the Area Chair for overseeing this process. As the discussion deadline approaches, we were wondering whether any further clarification would be helpful. We summarize the additions to our revised manuscript below.

1. Experiments

  • Best-of-N scaling (9MQk W1; NyDg W4). We added fixed-budget best-of-N comparisons for N = 2, 4, and 8 across 12 tasks. Branch-reset is higher than all three settings on 10/12 tasks (Appendix G).

  • Additional model and harness (9MQk W3; vrhd W1/Q1; NyDg W1/W4). We evaluated Qwen3.6-27B with Codex CLI on five tasks under an equal processed-token budget. Branch-reset achieves the highest IQM at 0.48 [0.37, 0.59], compared with 0.42 [0.34, 0.53] for successive halving, 0.35 [0.23, 0.49] for single-run, and 0.29 [0.20, 0.39] for best-of-4 (Section 5.4).

  • Additional controls (9MQk Q1; vrhd W3; NyDg W1–W3). We added controls isolating context reset from branching and diversity prompting, together with processed-token, equal-output-token, and equal-submission comparisons (Table 4; Appendices B, H, and J).

2. Analyses and Clarifications

  • Idea-space and round-level analyses (9MQk W1; vrhd W2/Q2/Q3; NyDg W3). We added additional analyses showing that resets induce larger idea-space moves than single-run trajectories and that nearly all mid-run rounds improve the best score (Appendices I and O).

  • Aggregate metrics (vrhd W3; NyDg W2). We added IQM 95% confidence intervals and relative raw-score gains. Branch-reset reaches 0.86 [0.83, 0.92], compared with 0.63 [0.44, 0.65] for best-of-4 and 0.51 [0.39, 0.58] for single-run (Table 1).

  • Novelty and task regimes (9MQk W2/Q2; NyDg W4). We clarified that our contribution is identifying and quantifying idea basins and empirically showing that branching with context reset (branch-reset) can help autoresearch agents escape them, rather than claiming either component is new. We also expanded Related Work and the task-regime analysis (Sections 6.2 and 7)

We hope our responses and the revised manuscript help clarify the reviewers’ concerns. We welcome any further comments or questions during the remaining days of the discussion period. Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.