- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a hierarchical self‑evolving multi‑agent system for AI‑driven science. It breaks high‑level scientific goals into DAG‑structured workflows via specialized agents. Two‑tier self‑evolution supports in‑task revision and cross‑task experience reuse as library assets. Tests over chemistry, biomedicine and materials show its performance rivals human expert teams. Limitations include short‑workflow bias and LLM hallucination risks.
Critical strengths
Existing scientific LLM‑agents only support single‑run execution without experience accumulation. This work integrates in‑flight task revision and cross‑task reusable human‑curated assets. Ablations verify both components are essential, delivering meaningful novelty to cut repeated human expert efforts, favouring acceptance.
Evaluations span chemistry, biomedicine and materials science, featuring expert correctness auditing, ablations, cross‑model transfer and controlled comparison against human teams. Unlike many papers relying on weak metrics, human validation confirms real scientific utility and strengthens acceptance arguments.
Critical weaknesses
Weakness 1
The evaluation only uses old, pre‑collected datasets instead of forward‑looking real‑world discovery tests. All feedback comes from published known measurements. There is no closed‑loop test where the system suggests brand‑new, untested candidates with unknown true results. Retrospective benchmarks work for early checks but cannot prove generalization to open‑ended discovery. The agent may fit statistical patterns in existing data rather than truly discovering new science. This brings uncertainty for real‑world use and hurts its case for ICLR acceptance.
Weakness 2
Testing is mostly limited to short research workflows. Few experiments target work with more than four scientific steps or frequent AI‑plan updates. Human final approval is required in this framework, yet the paper gives no numbers on actual human workload: review rounds, expert time cost or scaling bottlenecks with growing asset libraries. Real‑world science often uses long, branching multi‑stage pipelines. Unresolved scalability concerns lower confidence for complex real‑world research campaigns and harm acceptance prospects.
Questions for authors
-
Retrospective dataset evaluation a. What concrete technical safeguards stop agents from seeing hidden ground‑truth values (yields, energies) before they “request a measurement” in simulation? Please explain how leakage is prevented. b. If we add noise or out‑of‑distribution data to simulated results, how often will intra‑task evolution make wrong revision choices? Can you provide quantitative numbers?
-
Human‑in‑the‑loop oversight a. Across your benchmark tasks, what is the average number of human‑review actions (workflow / asset approvals) per single task in your existing experiments? b. As the asset library grows bigger, what filtering stops too many candidate assets going to human reviewers? Do you have measured data showing how review workload scales?
-
Intra‑task evolution ablation interpretation a. How much performance gain of full intra‑task evolution comes from the Review‑Chair LLM versus the two reviewer agents? Did you test an oracle ground‑truth Review‑Chair to spot LLM judgement errors?
-
Asset transfer and asset quality a. Before humans review generated assets: are there automatic rules to throw out low‑quality assets, or is human check the only filter? Please state the existing automated rejection criteria. b. In RQ5 cross‑model asset transfer: are assets fully model‑independent, or do they carry quirks from the original strong LLM? What asset parts stay intact across different LLMs?
-
Workflow DAG complexity limits What real‑world failures happen when building or updating multi‑stage, complex‑dependency DAG workflows? Does your execution harness have unreported hard‑coded caps for DAG depth or branching?