- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Branch-Reset, a harness intervention on a long horizon autoresearch agents. A run opens with K=4 agents starting from an empty workspace, and when 33% and 66% of the budget has been spent, the best-scoring agent is forked into K=4 new agents that inherit its workspace but start with an empty chat context. Each fork runs for about 0.06B credits, and the run continues from the highest-scoring fork. The authors reported consistent improvement over single-run and best-of-4 baselines on 13 tasks. An ablation on 11 tasks attributes the gain to both branching and context reset.
Critical strengths
- The problem is important and timely. The paper documents clearly that autoresearch runs plateau at persistently different levels. This observation is useful to the community.
- The empirical study is large and costly. It covers 13 open-ended tasks with individual runs of up to 107 hours and about 7M credits in total.
- The authors took great effort in defining and visualizing the “idea basins”, and I appreciate this effort.
- The appendix discloses all agent prompts and reports a negative result: plateau-triggered schedules rarely fired. This transparency is appreciated.
Critical weaknesses
W1. "Equal compute" is matched only on LLM tokens, which may systematically favor the reset conditions. Credits are a function of input and output tokens. An agent with a fresh context pays far fewer input tokens per step than one carrying a long history, even a compacted one. For the same credits, it can therefore take more actions and make more graded submissions. The paper reports no step counts, submission counts, or tokens per step for any condition.
W2. The improvement is concentrated in a few tasks, and the empirical improvement looks fragile.
(a) The variance of the results in Table 1 is high. In the 13 tasks, most tasks do not have significant improvement over baselines. On some tasks Branch-Reset is even worse than Single Run. The authors did not try to explain why this happened.
(b) The head line “66%” or “44%” improvement is computed on min-max normalized scores and does not make sense. It is like 20℃ is not twice as hot as 10℃
(c) The ablation is similarly fragile. The claim that branching and resetting are both needed is clearly supported on only four of the eleven tasks: mainly “AHC014” and “order-add”. These largest gaps come from the highest-variance cells (e.g., SD 29.35 on AHC014 with n=5).
W3. The basin-escape mechanism is not sufficiently supported.
(a) The only quantitative analysis of a jump covers a single round on polyomino (Fig. 3). Appendix G notes that many rounds produce no basin jump yet still improve the score, which suggests the gain does not require basin escape.
(b) The paper does not report, across all mid-run rounds, how often the selected branch lands in a new basin. Nor does it compare gains between rounds with and without a jump. A more detailed behaviour analysis is needed here.
W4. Methodological novelty is limited, and the baselines do not represent the closest prior work.
(a) Branching with selection is not new. As the related-work section itself acknowledges, it is go-with-the-winners, or the exploit step of Population Based Training.
(b) Context reset with a persistent workspace is not new either. Works like FunSearch and AlphaEvolve, both cited in the paper, build every LLM query from a fresh prompt that contains only the top-scoring programs.
(c) The only population baseline, best-of-4, is weak by construction. B is set where single runs plateau, so each B/4 run is cut off before it plateaus. Stronger baselines are needed to isolate the contribution, e.g., successive halving, an early best-of-4 followed by a single run, and periodic restore-to-best with a fresh context but no branching. The authors should also discuss recent work on population-based autoresearch (e.g., GEAR) and on context contamination when retrying.
Questions for authors
See the above weakness section. I will reconsider the ratings based on the authors' responses.
5 comments
Continued Response to Reviewer NyDg (Part 2/2)
Continuing from our previous response, we address the remaining weakness below.
W4: Novelty, Baselines, and Related Work
We agree with the reviewer that novelty and baseline positioning should be stated precisely. To address this point, we clarified the novelty claim, added stronger baselines, and expanded related work. We do not claim branching or context reset alone as new. The novelty is diagnosing context-induced stagnation in stateful long-horizon autoresearch agents and showing that workspace branching plus periodic context reset mitigates it. FunSearch and AlphaEvolve issue stateless, single-turn calls from stored programs, so no multi-turn agent history accumulates. Our agents run for hours to days in persistent workspaces with accumulated tool calls, logs, and reasoning, where keeping history is the default.
Both components matter, and the added baselines sharpen where the method helps:
| Task | best-of-2 | best-of-4 | best-of-8 | branch-reset |
|---|---|---|---|---|
| polyomino | 91.5 | 90.7 | 92.5 | 94.4 |
| job-shop | 9.92 | 9.91 | 9.85 | 10.33 |
| AHC014 | 27.0 | 20.7 | 21.4 | 57.5 |
| p150 | 122.6 | 123.5 | 124.5 | 129.7 |
| p171 | 93.3 | 88.9 | 93.0 | 92.8 |
| sphere-pack | 43.6 | 43.8 | 43.7 | 44.9 |
| relay-lay | 72.59 | 72.74 | 72.72 | 72.81 |
| LWE-rec | 149.6 | 141.1 | 116.2 | 144.6 |
| LLM-SQL | 26.93 | 26.93 | 27.00 | 27.04 |
| order-add | 25.5 | 26.2 | 28.2 | 49.5 |
| Borden-inv | 70.0 | 76.5 | 76.5 | 85.9 |
| kernel | 126.2 | 122.5 | 119.3 | 135.2 |
Branch-reset is higher than all three fixed-budget best-of-N settings on 10/12 tasks, though larger N also makes each independent run shallower. In the component ablation, single-run/reset-only/branch-only/branch-reset reach IQM 0.44 [0.31,0.50] / 0.57 [0.44,0.67] / 0.62 [0.52,0.70] / 0.80 [0.75,0.87]. In the Qwen suite, succ-halving reaches 0.42 IQM versus 0.48 for branch-reset; this is a smaller, mixed result with overlapping intervals, not a uniform win.
Reset-only tests fresh context without branching. For Qwen models, succ-halving is above single-run and best-of-4 in IQM and beats branch-reset on polyomino and AHC014, while branch-reset leads on p150 and p171 and the two tie on job-shop.
We expanded related work to discuss go-with-the-winners, Population Based Training, FunSearch, AlphaEvolve, GEAR, SwarmResearch, Heuresis, MLEvolve, and context contamination when retrying across turns. Branch-reset differs by keeping one lineage, preserving workspace state, and discarding conversational history at selected forks. We added the positioning in Related Work, the main baseline results in Table 1, the open-source baseline in Section 5.4, and fixed-budget best-of-N scaling in Appendix G.
Concluding Remarks
We thank the reviewer again for the detailed feedback and constructive suggestions. We added equal-output-token and equal-submission controls, processed-token open-weight evaluation, improved aggregate metrics, ablation confidence intervals, controlled context-reset tests, all-round basin-jump quantification, per-round analysis, stronger baselines, and expanded related work. We hope that our rebuttal was sufficient in addressing all the important questions raised by the reviewer and allows them to endorse our paper even more strongly with a potential score upgrade if they deem it appropriate.
Follow-Up on Our Response to Reviewer NyDg
Dear Reviewer NyDg,
We would sincerely appreciate hearing whether our response and new analyses addressed the questions you raised as the discussion deadline approaches. In brief, we added processed-token, equal-output-token, and equal-submission controls; reported IQM and ablation confidence intervals; quantified score-plateau recovery and mapped idea-space jumps; added fixed-budget best-of-N, reset-only, and Qwen successive-halving comparisons; and expanded our novelty and related-work discussion. Thank you again for your detailed feedback and for recognizing the importance, scale, and transparency of our work.
Response to Reviewer NyDg (Part 1/2)
Response to Reviewer NyDg (Part 1/2)
We thank the reviewer for the thorough feedback and for noting the problem's importance, the scale of the study, and the value of the idea-basin visualization. Due to character limits, we split our response into two comments. In this first comment (Part 1), we address W1-W3. Please see Part 2 for W4 and concluding remarks.
W1: Equal Compute, Tokens, and Submissions
We agree with the reviewer that compute matching should be carefully controlled. We added three controls. First, we evaluated Qwen3.6-27B under a processed-token budget, W = input - cached input + output:
Second, in the main credit-matched runs, branch-reset uses a median 1.18x output tokens and 1.46x graded submissions. Truncated at the median single-run total on each task, it still beats single-run on 5/6 tasks at equal output tokens and 7/11 tasks at equal submissions.
Third, extra submissions do not fully explain the gain: on sphere-pack, LWE-rec, and LLM-SQL, where branch-reset makes the most extra submissions, it gains under 3% or loses; on Borden-inv, it makes fewer submissions while gaining about 33%. The processed-token budget addresses cached context: cached history is not charged, so long context does not make single-run steps more expensive, while reset branches still pay to re-read workspace files. We added the processed-token evaluation in Section 5.4 and token/submission controls in Appendix H.
W2: Variance, Aggregate Metrics, and Ablation
We value the reviewer's comment regarding variance and aggregate metrics. We added directional tests, IQM, raw-score relative improvements, and a clearer explanation of where gains are largest. Branch-reset has a higher mean than single-run on 12/13 tasks and than best-of-4 on all 13. On the eleven GPT-5.6-terra tasks, 49/55 branch-reset runs exceed the task's median single-run score. It lowers raw standard deviation on 10/13 tasks; it rises on AHC014 and order-add because jumps raise the upper tail, and on LWE-rec.
The gain size tracks single-run spread. AHC014, tinyCIFAR, Borden-inv, and order-add have the widest spread and largest gains. Low-spread tasks such as relay-lay and LLM-SQL gain under 1%. LWE-rec is the only task where branch-reset trails single-run; its runs end in a narrow band and resets spend budget re-orienting.
We replaced normalized-ratio headlines with IQM and raw-score relative gains. Table 1 now reports IQM with 95% stratified-bootstrap intervals: 0.86 [0.83,0.92] for branch-reset, 0.63 [0.44,0.65] for best-of-4, and 0.51 [0.39,0.58] for single-run. On native raw scores, branch-reset improves over single-run and best-of-4 by averages of 26.7% and 24.2% and medians of 6.9% and 4.2%.
For ablations, branch-reset has the highest mean on 8/11 tasks and beats reset-only and branch-only on 9/11. Aggregate IQM rises from 0.44 [0.31,0.50] single-run to 0.57 [0.44,0.67] reset-only, 0.62 [0.52,0.70] branch-only, and 0.80 [0.75,0.87] branch-reset. The tinyCIFAR checkpoint test further isolates context reset: 15/20 reset branches escape score plateaus versus 0/20 branches that keep history. We added these metrics in Table 1 and Appendix A.
W3: Basin-Escape Mechanism and Round-Level Behavior
We agree that more behavior analysis strengthens the mechanism claim. We added both aggregate jump analysis and a controlled checkpoint test. Score gains do not require an idea-space basin jump in every round: branch-reset helps through fresh-context refinement near the inherited workspace and occasional large jumps. The controlled tinyCIFAR test measures score-plateau escape, not necessarily idea-space movement: clearing context breaks stagnation with 15/20 reset branches escaping versus 0/20 branches that retain history.
We added a jump analysis across the five mapped tasks: 90 valid mapped reset branches over 24 rounds. A large move is when a reset branch's first artifact is farther from its branch point than 99% of single-run steps after the first quarter of the budget; reset branches make such moves 9/90 times, and at least one branch jumps in 9/24 rounds. Jump sizes range from 12x to 153x the median single-run step.
A large-move branch is selected in 2/24 total mapped rounds, i.e. 2/9 jump rounds. This is expected because a branch that changes approach often starts from an untuned implementation. In both selected-jump rounds, however, it gives the largest single-round gain on its task. We added the controlled context-reset test in Appendix J and the mapped-jump analysis in Appendix O of the revised version.