- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ScienceFlow, a long-horizon autonomous research framework that integrates recoverable executable states, LLM-guided research trajectory selection (ESTRA), and evidence-aware execution control. It demonstrates strong performance across machine learning, scientific modeling, and mathematical optimization benchmarks.
Critical strengths
The framework effectively integrates state recovery, adaptive exploration, and resource management into a unified research workflow. Extensive benchmark evaluations demonstrate competitive performance, while ablation studies provide evidence for the effectiveness of ESTRA and execution control.
Critical weaknesses
The theoretical analysis emphasizes the advantages of LLM-based semantic decisions over validation-score-driven search methods such as MCTS and genetic algorithms. However, the experiments do not directly compare ESTRA against these alternatives under controlled settings, leaving this central theoretical motivation insufficiently validated.
Questions for authors
Can the authors provide a controlled comparison between ESTRA and validation-score-driven search strategies, such as MCTS or genetic algorithms, under identical computational budgets? This would help establish whether LLM-based semantic decisions offer the claimed advantages in practice.
A1: We thank the reviewer for the suggestion of a direct comparison under identical computational budgets. We add experiments (Table R6) comparing ESTRA with Random, UCT, and Evolve as alternative re-anchoring policies. All methods retain the same state recovery infrastructure and use the same backbone model, resources, seeds, and compute budget. We select the four tasks unmedaled in the original ScienceFlow runs on MLE-bench Lite and compare the performance of the Random, UCT, Evolve, and LLM (ESTRA) strategies.
Table R6: Controlled comparison of re-anchoring strategies on four previously unmedaled tasks (3 seeds).
The LLM (ESTRA) strategy achieves the best average rank (1.75) across the four tasks, providing practical evidence for the effectiveness of ESTRA's re-anchoring policy. No additional task-specific tuning of numerical search hyperparameters was performed.