Summary: This paper presents Continual AutoResearch, an evaluation environment for assessing whether autonomous research agents can learn and apply knowledge from previous tasks in machine learning. Unlike traditional AutoResearch evaluations, which create a new agent for each task, the proposed framework compares stateful agents (agents that keep track of their execution history, files, and previous research experience) to stateless agents (agents that start fresh for each task). The evaluation includes nine task families including synthetic AutoML, symbolic regression, time-series forecasting, and tabular AutoML. Five frontier agents are evaluated using three complementary metrics: first-solution quality, time-convergence efficiency, and final held-out performance. The results demonstrate that persistent execution increases research efficiency across all five agents, primarily by allowing for earlier generation of valid and competitive solutions. However, these increases in efficiency do not consistently lead to improved final outcomes. Additionally, memory ablations show that having relevant information does not necessarily lead to effective knowledge transfer. Therefore, the paper concludes that experience retention and effective experience utilization are two distinct capabilities in continual autonomous research.
Contributions:
- This paper develops a benchmark for studying cross-task adaptation in autonomous ML research, comparing persistent and reset agents under matched task streams, environments, and time budgets.
- The authors evaluate first-solution quality, time-convergence efficiency, and final held-out performance, providing a more detailed understanding of how previous experience affects subsequent research.
- The study evaluates five frontier agents across nine task families, demonstrating that persistent execution consistently increases search efficiency but has heterogeneous effects on final performance.
- Through memory ablations, candidate-matched comparisons, time-budget experiments, and trajectory case studies, the paper examines how retained experience affects initialization, exploration, and final solution selection.
Validation: The authors evaluate five frontier agents across nine task families, with 32 transfer-eligible tasks per agent. The stateful and stateless conditions use matched datasets, evaluation procedures, and per-task budgets, typically 30 minutes. The evaluation employs private held-out test sets and measures both research efficiency and final solution quality. The results show positive mean time-convergence gains for all five agents, ranging from +3.0 normalized reward points for GPT-5.4 to +21.3 for Kimi K3. However, final performance gains are substantially less consistent. Sonnet 4.6 achieves the largest improvement of +7.6 points, whereas Kimi K3 and GPT-5.4 show only marginal gains, and Opus 5 and GPT-5.6-Sol experience small declines. The authors additionally investigate the sources of these efficiency gains through candidate-count matching and first-solution latency controls. These experiments indicate that persistent agents primarily benefit from producing valid solutions earlier rather than consistently improving their subsequent optimization ability. Memory ablations comparing relevant, shuffled, and absent memory further show that task-relevant experience does not reliably improve final performance. Additional experiments examine different time budgets, stream lengths, and operational failure patterns.
Response to Reviewer
We thank the reviewer for their time. We respond to each point and would be glad to discuss further.
W1. Predictability and depth of the findings.
The reviewer states that the findings are 'entirely predictable.' We respectfully disagree. Even results that seem intuitive in hindsight need to be measured, and ours show where persistence helps and where it does not. Several of them go against what one would reasonably expect:
The reviewer also notes that our conclusions "provide no deeper insight, such as how to address the problems identified." For each failure we identify, the paper proposes a concrete fix: memories should record when they apply and be revised when contradicted (App. C.3, D.3; §6.2); agents should keep the best simple solution as the default and accept replacements only if they hold across validation splits (App. D.1–D.2); and because experience currently helps initialization but not later optimization (Table 10), continual agents need ways to use prior experience during search. We will clarify and collect these implications in §6.2 in the revision.
W2. Writing, structure and number of experiments.
On the number of experiments, our study includes:
On writing, we also review the clarity of our claims and writing. We detail our added changes in the General Response.
W3. Completeness of the literature review.
We have expanded the related-work section and added a comparison with five prior continual-agent benchmarks (General Response, point 3). We would be grateful if the reviewer could point us to specific works they consider missing, and we will gladly include them.