acceptodds
Under review as a conference paper at ICLR 2027

From AutoResearch to Continual AutoResearch: Do Agents Learn Across Tasks?

Abstract

Autoresearch has shown that agents can autonomously write code, run experiments, and iteratively improve machine learning solutions. Yet most evaluations start a new agent session from scratch for each new problem, making them costly and leaving unclear whether agents can retain and apply prior insights when adapting to new tasks. We introduce Continual AutoResearch, an open-ended environment that compares persistent and reset instances of the same agent under matched task streams, data, and time budget. Continual AutoResearch covers nine task families, spanning synthetic tasks, symbolic regression, time-series forecasting, and tabular AutoML. Across five frontier agents, we measure first-solution quality, research efficiency, and final held-out performance. To distinguish insight transfer from session continuity, we run memory ablations that vary what information agents retain from prior tasks. We show that persistent state improves research efficiency of all five agents, allowing them to find stronger solutions sooner. However, these gains do not consistently translate into better final solutions. Overall, our findings suggest that retaining experience and utilizing it effectively are distinct capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, automated AI R&D, continual learning, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mKUEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether an AI research agent, when performing multiple research tasks, can apply past experience to subsequent tasks and thereby improve over time. The authors found that retaining history generally enables the agent to produce a workable initial solution more quickly, though it does not necessarily improve the final outcome.

Critical strengths

This article conducts systematic research on the research question and has reached certain conclusions.

Critical weaknesses
  1. The research conclusions in this paper offer nothing of value to the community, as the findings are entirely predictable. They provide no deeper insight, such as how to address the problems identified.

  2. The writing is extremely poor. The pacing and structure are problematic, and it feels like padding. There are very few experiments, and the experimental analysis is unnecessarily lengthy.

  3. The author's literature review is highly incomplete, omitting many key works.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for their time. We respond to each point and would be glad to discuss further.

W1. Predictability and depth of the findings.

The reviewer states that the findings are 'entirely predictable.' We respectfully disagree. Even results that seem intuitive in hindsight need to be measured, and ours show where persistence helps and where it does not. Several of them go against what one would reasonably expect:

  • Efficiency improves, but outcomes do not. Persistence improves research efficiency for all five agents: they reach a valid solution sooner (Table 7), and time-convergence gains are positive across the board (Table 2). One would expect faster progress to lead to better final solutions, yet final outcomes do not reliably improve.
  • Memory relevance had little effect on final outcomes. One would expect relevant memory to beat unrelated memory. In the memory ablation (§5.5), shuffled cards taken from an unrelated stream gave final gains over no memory comparable to matched cards for Sonnet 4.6 and GPT-5.4. For Sonnet, matched cards improve the final outcome by +14.7 points over no memory, yet they are no better than shuffled cards (matched − shuffled = −0.2; Table 12). For GPT-5.4, the corresponding numbers are +5.0 and +0.2.
  • Stronger models benefit less. Within each harness, the model with lower reset performance gains more from persistence. Sonnet 4.6 gains +7.0 in initialization against +1.3 for Opus 5, and GPT-5.4 gains +3.0 against −1.0 for GPT-5.6-Sol (Table 2).
  • More experience is not always better. The efficiency advantage is front-loaded (+24.0 points at 25% of the budget, +1.2 at the deadline; Table 8) and shrinks with larger budgets (Fig. 3). Longer histories can even slow convergence (Appendix B.2).
  • Retained experience creates specific failure modes. Retained lessons can anchor agents to weaker model families, and strong early candidates can be replaced by overfitted ones (Appendices C and D).

The reviewer also notes that our conclusions "provide no deeper insight, such as how to address the problems identified." For each failure we identify, the paper proposes a concrete fix: memories should record when they apply and be revised when contradicted (App. C.3, D.3; §6.2); agents should keep the best simple solution as the default and accept replacements only if they hold across validation splits (App. D.1–D.2); and because experience currently helps initialization but not later optimization (Table 10), continual agents need ways to use prior experience during search. We will clarify and collect these implications in §6.2 in the revision.

W2. Writing, structure and number of experiments.

On the number of experiments, our study includes:

  • 5 frontier agents × 9 task families (32 paired transfer tasks per agent);
  • three repeated runs of the full benchmark for GPT-5.6-Sol and, newly, for Sonnet 4.6;
  • three time budgets;
  • stream-length and candidate-matched controls;
  • a three-condition memory ablation;
  • new in this rebuttal: task-order controls for two agents and a normalization and weighting sensitivity analysis (see the General Response).

On writing, we also review the clarity of our claims and writing. We detail our added changes in the General Response.

W3. Completeness of the literature review.

We have expanded the related-work section and added a comparison with five prior continual-agent benchmarks (General Response, point 3). We would be grateful if the reviewer could point us to specific works they consider missing, and we will gladly include them.

We thank the reviewer again for the comments. As the discussion period is nearing its end, we would be happy to clarify and answer any remaining questions.

If we have addressed all your concerns, we would be very grateful if you would consider raising your score.

Best.

Thank you for the response. I still find the main findings largely expected, and the additional experiments do not change this assessment. I will keep my score.

Reviewer_JpiXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether autoresearch agents benefit from retaining experience across sequential machine learning tasks compared to starting with a fresh instance for each task. The authors introduce a continual research framework to evaluate this where the framework spans 9 ML tasks. They evaluate 5 different models as agents and measure their performance on the framework with and without persistent memory. They find that memory helps agents find good solutions faster but doesn’t produce substantially better final solutions.

Critical strengths

The paper evaluates multiple different strong recent baselines like Kimi K3 and 5.6 Sol in an attempt to evaluate continual learning in an autoresearch setting which are both important topics.

Critical weaknesses

The main evaluation is only a run across a fixed set of tasks per agent which is insufficient to demonstrate the usefulness of the benchmark or to substantiate the claims in the contributions.

The paper mentions several continual learning benchmarks in section 2.2 but doesn’t mention how this framework provides a substantial novel addition that is lacking from previous benchmarks. Many previous benchmarks already tackle continual learning of agents performing research, specifically ‘AgentCL’ and ‘Continual Learning Bench’ seem to cover many of the points mentioned in this paper.

Figure 9 shows all 5 agents achieving approximately 0.8 across both stateful and stateless modes, this to me indicates a concern that the task all have a single simple sufficient solution that all agents are able to find within the 30 minutes. Otherwise I would not expect stateful sonnet 4.6 to match the abilities of sol 5.6 on difficult tasks.

Questions for authors

Regarding the continual autoresearch framework, what novel insights does it provide? Why do you believe that knowledge about synthetic shared latent tasks translates to time-series forecasting given that these agents have access to scientific research papers indexed by google about many niche tasks. I would not expect knowledge about these distinct tasks to provide an edge on the other tasks over the available literature.

Could you provide a figure where the x-axis and lines are figure 10 but the y-axis is from figure 9. Does gpt 5.6 sol perform significantly worse than sonnet 4.6 in the tasks?

Could you provide historical logs of agent chats along with evaluations? This doesn’t require any code, only text input/output and json files for the task evaluations.

3 comments
AuthorsOpenReview ↗

Response to Reviewer JpiX

We thank the reviewer for their feedback and for concrete requests that we could act on. Several concerns stem from places where our presentation was unclear. We have revised those passages and give the evidence below.

W1: Single run over a fixed set of tasks.

Section 5.1 and Appendix Figure 4 report three repeated matched comparisons for GPT-5.6-Sol, in which outcome and initialization gains varied by at most 0.75 points; Figure 3 also uses three repetitions per budget. We have added two further runs of the full Sonnet 4.6 experiments and task-order controls for Opus 5 and GPT-5.6-Sol (General Response, point 1)

W2: Novelty relative to AgentCL and Continual Learning Bench.

We agree these are the closest works and have expanded the comparison (General Response, point 3). We also refer to the first point of the General Response about the novelty and contributions.

W3: Figure 9 suggests the tasks have a single simple solution.

Most agents do reach a reasonable solution within the 30-minute budget, but the tasks do not share one simple solution. In the reset condition, agents differ by about 10 points (70.3 to 80.5 normalized points; Table 2), and transfer varies widely across task families (Figure 8). Our focus is also on the research process rather than only the final outcome. Figures 2 and 9 show that agents start well below their final level and improve throughout the budget, and persistence changes how quickly they get there: stateful runs lead by +24.0 points at 25% of the budget, versus +1.2 at the deadline (Table 8). Regarding Sonnet, stateless Sonnet 4.6 reaches a mean outcome reward of 70.3 normalized points, against 80.5 for GPT-5.6-Sol. Sonnet approaches GPT-5.6-Sol only when stateful (77.9 vs. 79.7), which reflects its large transfer gain rather than a shared performance ceiling.

Q1: Novel insights, and transfer from synthetic tasks to forecasting.

Our work attempts to study the adaptive behavior of current agents in autoresearch settings. We showed that persistent experience changes how agents do research more than what they produce in the end. Under matched budgets, all five stateful agents reach good solutions faster than their reset counterparts (time-convergence gains of +3 to +21 points). Final outcomes barely change for most agents (−0.9 to +0.4 points for four of five), because reset agents catch up through within-task search. Transfer is strongly agent- and task-dependent: negative transfer occurs on 33–66% of transfer tasks, and the weakest agent (Sonnet 4.6) benefits most (+7.6 outcome points). The benefit also depends on the setting. As time budgets grow, outcome gains shrink and initialization gains grow, and gains do not grow consistently with stream length. Finally, our insight intervention shows that agent-written insights from matched prior tasks do not consistently outperform size-matched shuffled insights. This suggests that current agents gain mostly from operational and session-level continuity rather than from portable scientific knowledge. Together, our results argue that research-agent evaluations should treat persistence as an explicit intervention with matched reset controls, and should measure search efficiency and negative transfer, not only final scores.

We first clarify our setup, because we neither test nor claim transfer between families. Each stream contains tasks from a single family (for example, five shared-latent tasks, or four or five related forecasting datasets), and the agent's state is reset between streams. What can transfer is family-specific structure: a recurring latent representation, a useful change of variables, or seasonal periods and context lengths that need recalibrating for a related dataset. The synthetic and symbolic-regression tasks are generated by us, so no literature on them exists. For public datasets, agents receive anonymized task and dataset identifiers (App. A.1), to prevent them from recognizing the benchmark and applying a known solution. Agents in both conditions have the same tool and web access, so any benefit from the literature is shared by stateful and reset runs and cancels out in our paired gains. On novel insights: persistent experience changes how agents do research more than what they produce in the end.

AuthorsOpenReview ↗

Response to Reviewer (2/2)

Q2: Best hidden reward by task position.

We added the requested figure (Figure 12). It uses the axes and task-position lines of Figure 10, with absolute mean best hidden reward on the y-axis as in Figure 9, shown separately for stateful and stateless runs.

No, GPT-5.6-Sol does not perform worse than Sonnet 4.6. Over the 32 transfer tasks, Sol achieves a higher family-balanced outcome reward in both settings: 80.5 vs. 70.3 normalized points stateless and 79.7 vs. 77.9 stateful. Paired on the 31 transfer tasks both agents share, Sol is better on 29 tasks in the stateless setting (Wilcoxon p<0.001p < 0.001) and on 23 in the stateful setting (p=0.01p = 0.01). The figure shows the same pattern on the synthetic streams: stateless Sonnet improves slowly, while stateful Sonnet reaches high reward early on later tasks. Sonnet thus approaches GPT-5.6-Sol only when stateful, which reflects its large transfer gain rather than weaker performance by GPT-5.6-Sol.

Q3: Release of agent logs and evaluation files.

Agent traces will be shared alongside the camera-ready version, once they are anonymized.

We thank the reviewer again for the comments. As the discussion period is nearing its end, we would be happy to clarify and answer any remaining questions.

If we have addressed all your concerns, we would be very grateful if you would consider raising your score.

Best.

Reviewer_omegOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper presents Continual AutoResearch, an evaluation environment for assessing whether autonomous research agents can learn and apply knowledge from previous tasks in machine learning. Unlike traditional AutoResearch evaluations, which create a new agent for each task, the proposed framework compares stateful agents (agents that keep track of their execution history, files, and previous research experience) to stateless agents (agents that start fresh for each task). The evaluation includes nine task families including synthetic AutoML, symbolic regression, time-series forecasting, and tabular AutoML. Five frontier agents are evaluated using three complementary metrics: first-solution quality, time-convergence efficiency, and final held-out performance. The results demonstrate that persistent execution increases research efficiency across all five agents, primarily by allowing for earlier generation of valid and competitive solutions. However, these increases in efficiency do not consistently lead to improved final outcomes. Additionally, memory ablations show that having relevant information does not necessarily lead to effective knowledge transfer. Therefore, the paper concludes that experience retention and effective experience utilization are two distinct capabilities in continual autonomous research.

Contributions:

  1. This paper develops a benchmark for studying cross-task adaptation in autonomous ML research, comparing persistent and reset agents under matched task streams, environments, and time budgets.
  2. The authors evaluate first-solution quality, time-convergence efficiency, and final held-out performance, providing a more detailed understanding of how previous experience affects subsequent research.
  3. The study evaluates five frontier agents across nine task families, demonstrating that persistent execution consistently increases search efficiency but has heterogeneous effects on final performance.
  4. Through memory ablations, candidate-matched comparisons, time-budget experiments, and trajectory case studies, the paper examines how retained experience affects initialization, exploration, and final solution selection.

Validation: The authors evaluate five frontier agents across nine task families, with 32 transfer-eligible tasks per agent. The stateful and stateless conditions use matched datasets, evaluation procedures, and per-task budgets, typically 30 minutes. The evaluation employs private held-out test sets and measures both research efficiency and final solution quality. The results show positive mean time-convergence gains for all five agents, ranging from +3.0 normalized reward points for GPT-5.4 to +21.3 for Kimi K3. However, final performance gains are substantially less consistent. Sonnet 4.6 achieves the largest improvement of +7.6 points, whereas Kimi K3 and GPT-5.4 show only marginal gains, and Opus 5 and GPT-5.6-Sol experience small declines. The authors additionally investigate the sources of these efficiency gains through candidate-count matching and first-solution latency controls. These experiments indicate that persistent agents primarily benefit from producing valid solutions earlier rather than consistently improving their subsequent optimization ability. Memory ablations comparing relevant, shuffled, and absent memory further show that task-relevant experience does not reliably improve final performance. Additional experiments examine different time budgets, stream lengths, and operational failure patterns.

Critical strengths
  1. The paper addresses a significant gap in autonomous research evaluation by examining whether agents can benefit from experience accumulated across tasks rather than starting from scratch each time. This is relevant to the development of persistent research assistants and long-running autonomous research systems.

  2. Comparing stateful and stateless instances of the same agent on identical task streams provides a useful experimental design for measuring the overall effect of persistence. The use of private held-out evaluation also reduces the risk that observed improvements result from direct optimization against test scores.

  3. The distinction between first-solution quality, time-convergence, and final performance is particularly valuable. The additional candidate-count and latency controls help explain why persistent agents improve efficiency without necessarily achieving better final solutions.

  4. Rather than focusing only on improvements, the paper examines negative transfer, memory-induced anchoring, weak follow-up exploration, and incorrect final solution selection. These analyses provide valuable insights into the limitations of current continual research agents.

Critical weaknesses
  1. Stateful execution simultaneously retains previous code, transcripts, files, and execution context. Therefore, improvements may result from reusing implementations or avoiding initialization overhead rather than learning transferable research strategies. Although the paper includes explicit-memory ablations, these are restricted to a small subset of tasks and do not fully isolate the contributions of different retained-state components.

  2. The robustness analysis shows that much of the time-convergence advantage disappears after controlling for first-solution latency and candidate counts. After additionally accounting for first-solution quality, no agent demonstrates a positive aggregate advantage in subsequent optimization. This narrows the interpretation of the findings, as the results primarily establish faster initialization rather than improved research exploration or optimization from accumulated experience

  3. Sonnet 4.6 is the only agent showing a substantial mean final-performance improvement, but its median gain is significantly smaller (+1.3 versus +7.6 points). Much of its average improvement comes from symbolic regression and energy forecasting, where gains are +41.2 and +18.5 points, respectively. This concentration raises concerns about whether the observed outcome benefits generalize across task families.

  4. The agents are evaluated using different native CLI harnesses, making cross-agent differences difficult to attribute solely to model capabilities.

Questions for authors
  1. Are the results dependent on the order in which the tasks were presented? To what extent would randomizing the order of tasks or interspersing unrelated tasks help to identify true knowledge transfer from general session persistence?

  2. While Appendix B.2 studies stream length in a limited budget, do the efficiency gains remain valid over much longer and more varied task sequences, especially as context accumulation and memory compaction become relevant? Do the efficiency gains continue to exist when comparing stateful and stateless execution with equal token or computational budgets as opposed to only wall clock time?

  3. How sensitive are the aggregate time-convergence gains to different normalization functions for RMSE and R2 and different task-family weighting schemes?

  4. Is the performance improvement demonstrated by Sonnet 4.6 (mean final-outcome gain of +7.6 points, median gain of +1.3 points) consistent when removing specific task families or outlier results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.