acceptodds
Under review as a conference paper at ICLR 2027

ScienceFlow: Executable-State Re-Anchoring for Long-Horizon Autonomous Research

Abstract

Long-horizon LLM agents can improve research outcomes, but their gains often rely on expensive model calls and poorly targeted computation. We introduce ScienceFlow, an end-to-end auto-research framework designed to convert limited inference and execution budgets into sustained research progress. ScienceFlow organizes research around recoverable executable states, allowing validated artifacts and intermediate results to persist across iterations. Its Executable-State Transition with Re-Anchoring (ESTRA) mechanism jointly selects an execution anchor from the current or archived states and an extend-or-redirect research direction, enabling continued exploration or redirection from recoverable executable states. An evidence-aware execution controller further admits and monitors jobs according to remaining budget, resource availability, and validated progress. We evaluate ScienceFlow across machine learning, scientific modeling, and mathematical optimization. On the full 75-task MLE-bench, ScienceFlow with DeepSeek-V4-Flash-Preview achieves a Any-Medal rate under a 24-hour budget per task run, evaluated over three independent benchmark sweeps. ScienceFlow also achieves a low mean API expenditure of $76.4 per benchmark sweep under our cost-accounting protocol. The source code is temporarily available in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
auto-research, LLM, Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EzB8OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ScienceFlow, a long-horizon autonomous research framework that integrates recoverable executable states, LLM-guided research trajectory selection (ESTRA), and evidence-aware execution control. It demonstrates strong performance across machine learning, scientific modeling, and mathematical optimization benchmarks.

Critical strengths

The framework effectively integrates state recovery, adaptive exploration, and resource management into a unified research workflow. Extensive benchmark evaluations demonstrate competitive performance, while ablation studies provide evidence for the effectiveness of ESTRA and execution control.

Critical weaknesses

The theoretical analysis emphasizes the advantages of LLM-based semantic decisions over validation-score-driven search methods such as MCTS and genetic algorithms. However, the experiments do not directly compare ESTRA against these alternatives under controlled settings, leaving this central theoretical motivation insufficiently validated.

Questions for authors

Can the authors provide a controlled comparison between ESTRA and validation-score-driven search strategies, such as MCTS or genetic algorithms, under identical computational budgets? This would help establish whether LLM-based semantic decisions offer the claimed advantages in practice.

1 comment

Q1: Can ESTRA be compared with validation-score-driven search (e.g., MCTS, genetic algorithms) under identical budgets?

A1: We thank the reviewer for the suggestion of a direct comparison under identical computational budgets. We add experiments (Table R6) comparing ESTRA with Random, UCT, and Evolve as alternative re-anchoring policies. All methods retain the same state recovery infrastructure and use the same backbone model, resources, seeds, and compute budget. We select the four tasks unmedaled in the original ScienceFlow runs on MLE-bench Lite and compare the performance of the Random, UCT, Evolve, and LLM (ESTRA) strategies.

Table R6: Controlled comparison of re-anchoring strategies on four previously unmedaled tasks (3 seeds).

MethodDog BreedNYC TaxiRANZCRTPS-May
MetricLog Loss ↓RMSE ↓Mean AUC ↑AUC ↑
Random0.25185 ± 0.091273.81391 ± 0.008160.93762 ± 0.016710.99659 ± 0.00051
UCT0.28553 ± 0.067594.42730 ± 0.785360.91288 ± 0.051110.99714 ± 0.00077
Evolve0.27636 ± 0.150214.01019 ± 0.226700.92470 ± 0.049390.99658 ± 0.00095
LLM (ESTRA)0.23485 ± 0.149163.78745 ± 0.012550.91727 ± 0.038240.99669 ± 0.00052

The LLM (ESTRA) strategy achieves the best average rank (1.75) across the four tasks, providing practical evidence for the effectiveness of ESTRA's re-anchoring policy. No additional task-specific tuning of numerical search hyperparameters was performed.

Reviewer_AK6fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScienceFlow, an autonomous research agent composed of recoverable executable states (where each checkpoint is composed of a workspace snapshot, research memory, validation evidence, and resource record), re-anchoring, and an execution controller that monitor jobs according to the budget. This system achieve 70.22 ±1.18% Any-Medal rate under a 24-hour budget per task run and $76.4 per benchmark sweep.

Critical strengths
  • The proposed method has low reported API cost: In Figure 1, ScienceFlow with Deepseek-V4-Flash-Preview reaches very high performance among the baselines while being the cheapest.

  • Multiple case studies and ablations. The authors conducted case studies and ablations in the long-horizon setting and ablations on components they proposed in the agent, making the experiment more rigorous.

Critical weaknesses
  • Ablation: The ablation of w/o ESTRA removes all re-anchoring, which would very likely lead to performance degradation. A more reasonable ablation is to replace LLM re-anchoring with UCB/MCTS/random re-anchoring.

  • Unequal comparison between baselines: Compared to the baselines, ScienceFlow’s budget, hardware, and backbone models are different. It might be useful to include a baseline with a different harness + the same backbone model (DeepSeekV4Flash), or run ScienceFlow with a different backbone model (e.g., a Claude/GPT/Gemini variant as in the baselines)

  • I would appreciate the authors make the difference between this work and previous work more apparent in the Related Work.

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Response for weakness 1 and 2: more ablations

W1: The w/o ESTRA ablation removes all re-anchoring; a fairer ablation replaces LLM re-anchoring with UCB/MCTS/random strategies.

A1: We thank the reviewer for the suggestion. We compare alternative re-anchoring policies while retaining the same state recovery infrastructure, backbone, resources, seeds, and budget. We use the four tasks unmedaled in the original ScienceFlow runs on MLE-bench Lite, with a 12h budget, 3 seeds, on the same RTX 4090 hardware, and compare the Random, UCT, Evolve, and LLM strategies. The results are shown in Table R3.

Implementation. The fixed strategies select re-anchoring start points from the qualified stages saved by each worker and check after each stage whether a decision is triggered. The LLM strategy follows ESTRA's original candidate scope and trigger conditions.

  • UCT (an MCTS variant) uses UCB to select start points along the stage tree, progressive widening to control tree expansion, and UCB again to choose between continuing the current direction or switching; validation rewards from actual runs are backpropagated along the tree.
  • Evolve (a genetic-algorithm-style strategy) randomly samples up to 2 of the most recent 4 valid stages and starts from the one with the better validation score. Continuing vs. switching is chosen with 50% probability each.

Table R3: Controlled comparison of re-anchoring strategies on four previously unmedaled tasks (3 seeds).

MethodDog BreedNYC TaxiRANZCRTPS-May
MetricLog Loss ↓RMSE ↓Mean AUC ↑AUC ↑
Random0.25185 ± 0.091273.81391 ± 0.008160.93762 ± 0.016710.99659 ± 0.00051
UCT0.28553 ± 0.067594.42730 ± 0.785360.91288 ± 0.051110.99714 ± 0.00077
Evolve0.27636 ± 0.150214.01019 ± 0.226700.92470 ± 0.049390.99658 ± 0.00095
LLM (ESTRA)0.23485 ± 0.149163.78745 ± 0.012550.91727 ± 0.038240.99669 ± 0.00052

The LLM strategy achieves the best average rank (1.75). No additional task-specific tuning of numerical search hyperparameters was performed.


W2: Budget, hardware, and backbones differ across methods; a same-backbone different-harness baseline or ScienceFlow with a different backbone would help.

A2: We address this through two complementary comparisons.

(a) Same backbone, different harness. Table R4 presents the results. With the same deepseek-v4.1-flash backbone, hardware, and 12h budget, ScienceFlow achieves 77.27% vs. MLEvolve's 69.70%, indicating that the gains come from the framework rather than the backbone.

Table R4: Performance and LLM usage on MLE-bench Lite.

MetricMLEvolveScienceFlow
Code Versioncommit 9c5c8a3 (2026-09-04)Submitted code archive
Any Medal Rate (MEAN ± SEM) ↑69.70% ± 3.0377.27% ± 2.62
Medals per seed16 / 16 / 1417 / 18 / 16
LLM Cost ($) ↓24.03 ± 5.1112.22 ± 1.79
Token (M)52.38 ± 15.33397.89 ± 6.33
Output Token (M)23.10 ± 5.437.02 ± 0.47
Cache Hit Rate26.33% ± 1.1893.21% ± 0.85

(b) ScienceFlow with a different backbone. Table R5 compares harnesses with open models. ScienceFlow with Qwen3.6-27B, which has fewer total parameters, achieves the highest mean medal rate among the listed configurations (53.03%).

Table R5: Harness comparison with open models on MLE-bench Lite.

ModelHarnessHardwareAny Medal Rate (%) ↑Report Date
Qwen3.6-35B-A3BOpenMLE-Evo [R1]1 × RTX 4090 (12GB)39.392026-07-30
Qwen3.6-35B-A3BMLExpert Harness [R2]1 × H20037.88 ± 5.462026-10-15
Qwen3.6-35B-A3BClaude Code [R2]1 × H20042.42 ± 4.012026-10-15
Qwen3.6-27BScienceFlowRTX 4090 / L4053.03 ± 3.03This work

Due to API cost constraints, we did not switch to Claude/GPT/Gemini backbones; instead, we use the open Qwen3.6 as an alternative backbone. The results above show the framework is not tied to one specific model.

For the Qwen3.6-27B run, the gap relative to the deepseek-v4.1-flash backbone is concentrated on MLSP-2013-Birds, Spooky Author Identification, APTOS 2019, and Leaf Classification (3 medals each with deepseek-v4.1-flash vs. 0 with Qwen), with smaller drops on SIIM-ISIC, Denoising Dirty Documents, and the two text-normalization tasks.

References

[R1] Yang, J., et al. Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering. arXiv:2607.28568, 2026.

[R2] Du, S., et al. MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering. arXiv:2610.22344, 2026.

AuthorsOpenReview ↗

Response for Weakness3: Related Work

W3: I would appreciate the authors make the difference between this work and previous work more apparent in the Related Work.

A3: We have revised Section 4 (Related Work) to clarify how ScienceFlow differs from prior work in recoverable executable states, evidence-guided re-anchoring, and resource-aware execution control. The added discussion is highlighted in blue on page 9 of the revised manuscript.

Reviewer_7nNJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ScienceFlow is a framework for autonomous research that combines recoverable executable states, research-direction selection, memory management, and resource-aware execution control. Its ESTRA mechanism selects whether to continue from the current or an archived state and whether to extend or redirect the research trajectory. Experiments cover MLE-bench, scientific modeling, mathematical optimization, and KTTSP, with a reported 70.22% Any-Medal rate across three complete MLE-bench evaluations.

Critical strengths

This work offers solutions to the practical challenges inherent in autonomous agents designed for long-duration scientific research. Given that elements such as code, checkpoints, intermediate features, and experimental records are vital to the continuity of research, this paper proposes a unified management framework that defines these components as "recoverable states."

Critical weaknesses
  1. The models and budgets in Table 1 are not aligned; ScienceFlow uses DeepSeek-V4-Flash-Preview, while other methods employ Gemini, Claude, and various versions of DeepSeek. Furthermore, a footnote in Table A5 explicitly states that the strongest baseline—MLEvolve paired with Gemini-3.1-Pro-Preview—is allocated a 12-hour budget, whereas ScienceFlow is allocated 24 hours. Consequently, the current comparison is insufficient to determine the extent to which the improvements are attributable to the ScienceFlow mechanism itself. I cannot determine whether the performance is due to the model itself being strong or to the strength of ScienceFlow.
  2. The experiments rely on MLE-Bench, a benchmark that has largely fallen out of favor. Current research tends to focus on benchmarks such as PostTrainBench, MLS-Bench, SWE-Marathon, AutoResearchExam, and RSI_Exam.
Questions for authors

See weaknesses.

3 comments

W1: Models and budgets in Table 1 are not aligned (ScienceFlow 24h vs. MLEvolve 12h), so the source of the improvement is unclear.

A1: We thank the reviewer for highlighting this. To separate framework gains from backbone effects, we rerun the comparison with the same deepseek-v4.1-flash backbone and a 12h per-task budget for both systems (Table R1). Over three seeds on the 22 MLE-bench Lite tasks, ScienceFlow outperforms MLEvolve on every seed (17/18/16 vs. 16/16/14 medaled tasks; mean medal rate 77.27% vs. 69.70%), at about half the LLM cost ($12.22 vs. $24.03 per round).

Table R1: Performance and LLM usage on MLE-bench Lite. Both columns use the same backbone, hardware, and budget for a matched comparison.

MetricMLEvolveScienceFlow
Code Versioncommit 9c5c8a3 (2026-09-04)Submitted code archive
Any Medal Rate (MEAN ± SEM) ↑69.70% ± 3.0377.27% ± 2.62
Medals per seed16 / 16 / 1417 / 18 / 16
LLM Cost ($) ↓24.03 ± 5.1112.22 ± 1.79
Token (M)52.38 ± 15.33397.89 ± 6.33
Output Token (M)23.10 ± 5.437.02 ± 0.47
Cache Hit Rate26.33% ± 1.1893.21% ± 0.85

Note: MLEvolve is evaluated with the Sep 4 code (commit 9c5c8a3), keeping its default three parallel search branches and cold-start knowledge base; ScienceFlow uses the exact code archive submitted with our paper.

The advantage comes from four tasks: Jigsaw (2 vs. 0 medals), SIIM-ISIC (1 vs. 0), Aerial Cactus (3 vs. 2), and Random Acts of Pizza (3 vs. 2) — five additional medals over three seeds, i.e., a 7.58% gain in mean medal rate.

On the cost side, ScienceFlow uses more total tokens (397.89M vs. 52.38M) but achieves a higher cache hit rate (93.21% vs. 26.33%) and lower estimated LLM cost ($12.22 vs. $24.03 per round).


W2: MLE-Bench has largely fallen out of favor; evaluations on recent benchmarks (e.g., PostTrainBench, MLS-Bench, SWE-Marathon) are preferred.

A2: With Qwen3-1.7B-Base as the trained model, the same deepseek-v4.1-flash drives both Claude Code 2.1.219 and ScienceFlow, under identical H200 resources and a 10h budget. Details are presented in Table R2.

Table R2: PostTrainBench results with Qwen3-1.7B-Base under a 10h budget.

HarnessAIME 2025ArenaHard-WritingGPQA MainHealthBench-EasyHumanEvalGSM8K
Claude Code 2.1.2196.6726.826.1911.2160.1668
ScienceFlow6.6733.2728.7225.4952.4472.33

ScienceFlow leads on four of the six evaluations, ties on one, and trails on one, which provides additional evidence of its applicability to model post-training. We also note that MLE-Bench remains actively used by recent MLE-agent systems [R1, R2]. We prioritized PostTrainBench as a representative testbed for post-training research; evaluation on MLS-Bench and SWE-Marathon remains future work.

On HumanEval, ScienceFlow delivered its best recorded candidate: an early LoRA SFT model, whose dev score rose from 24.67% to 53.33% (development set; final evaluation: 52.44% in Table R2) after fixing EOS and decoding; later stages with full-parameter SFT and additional self-generated data did not yield a stronger candidate within the 10h budget, so the early LoRA model remained the final choice. Claude Code instead used the same budget to iterate on execution-verified answers, doctest augmentation, and checkpoint/decoding comparisons. These trajectories suggest different uses of the available budget, but do not isolate the cause of the performance gap.

References

[R1] Yang, J., et al. Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering. arXiv:2607.28568, 2026.

[R2] Du, S., et al. MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering. arXiv:2610.22344, 2026.

Thank you for the response. It addresses my concern. I have another question why can ScienceFlow be both cheaper and better? Could the authors elaborate more on this?

Dear Reviewer 7nNJ,

We sincerely thank you for this thoughtful follow-up. ScienceFlow combines reuse of research artifacts and evidence-guided re-anchoring with a high input-cache hit rate and fewer output tokens.

Why better. ESTRA picks an execution anchor and an extend-or-redirect direction from accumulated evidence and remaining resources; recoverable states let new branches reuse saved code, weights, and features rather than repeat implementation or training. Two examples:

  • Statoil Iceberg (paper case study). After CNN-only exploration stalls, ScienceFlow restores S59 and switches to tree models over saved CNN features—S112 trains LightGBM on S53's CNN logits and angle features, and S59 merges fold checkpoints from S02/S53—improving validation loss from 0.1100 to 0.0597 without retraining the CNN.
  • PostTrainBench HumanEval. Reusing a saved LoRA checkpoint and fixing only EOS/decoding raised the development score from 24.67% to 53.33% without retraining.

Why cheaper, and how cost is computed. We estimate LLM cost from reported API usage using DeepSeek's official pricing: cost = (input − cached) x 0.30/M + cached x 0.006/M + output x 1.20/M , with off-peak hours at half price. Cached input is thus 50× cheaper than uncached input. ScienceFlow actually uses more total tokens, averaging 390.87M input tokens per round, but achieves a mean input-cache hit rate of 93.21% across the three seeds. It generates only 7.02M output tokens per round vs. MLEvolve's 23.10M—output being the most expensive component. The resulting estimated cost is USD 12.22 vs. USD 24.03 per round.

Thus, higher performance and lower estimated LLM cost are compatible: artifact reuse supports continued exploration, while the observed cache hits and lower output-token usage reduce API cost under the stated pricing.

Sincerely,

Authors of ScienceFlow

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal

Dear PCs, SACs, and ACs,

We thank the reviewers for their constructive feedback. ScienceFlow combines recoverable executable research states, evidence-guided re-anchoring, and resource management to sustain long-horizon research. To further validate ScienceFlow, additional experiments on performance attribution, search strategies, and benchmark coverage are included in this rebuttal. The key results are presented below.

Performance and cost. With the same deepseek-v4.1-flash backbone and a 12h per-task budget, ScienceFlow outperforms MLEvolve on each of the three seeds on MLE-bench Lite (mean medal rate 77.27% vs. 69.70%). The LLM cost per round is about half that of MLEvolve ($12.22 vs. $24.03).

Search-strategy comparison. On four previously unmedaled tasks, we compare ESTRA with Random, UCT, and Evolve as alternative re-anchoring policies while retaining the same state recovery infrastructure. ESTRA achieves the best average rank (1.75). No additional task-specific tuning of numerical search hyperparameters was performed.

Broader applicability. We add experiments on PostTrainBench and with the open Qwen3.6-27B model. On PostTrainBench, with the same backbone, hardware, and 10h budget, ScienceFlow outperforms Claude Code on four of six evaluations, ties on one, and trails on one.

In summary, the additional experiments strengthen the evidence for ScienceFlow's effectiveness, cost efficiency, and applicability across models and research tasks.

To support verification of the reported results, we include the complete experimental material in the supplementary package (iclr_supplementary.zip): raw execution traces of both systems on all 132 MLE-bench Lite runs (full LLM trajectories and framework events, organized by seed and task), the official per-run results (results_summary.csv), and a README documenting the structure, budget setup, and anonymization.

Sincerely,

Authors of ScienceFlow

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.