- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Evidence-to-Action (E2A), a framework for iterative materials discovery that integrates historical search experience and retrieved external knowledge to guide subsequent search decisions. The framework uses two learned policies: a Workflow Policy that selects an evidence-use workflow for each task, and an Evolution Policy that dynamically chooses among exploration, refinement, and escape actions based on the search state. The authors evaluate E2A on 14 materials-design tasks and report improvements in hit rate and stable-success rate over independent generation. Additional experiments compare fixed workflows and learned versus rule-based search controllers.
Critical strengths
The paper addresses a meaningful problem in iterative materials discovery: how to translate accumulated search experience and external knowledge into effective search decisions. The hierarchical design separating workflow selection from search-action selection is reasonable.
The evaluation covers 14 materials-design tasks and multiple LLM backbones. The reported improvements in material discovery performance are encouraging, and the authors provide additional workflow and controller comparisons to investigate the contributions of different components.
Critical weaknesses
The proposed framework mainly combines existing techniques, including experience-based feedback, retrieval-augmented generation, offline policy learning, and evolutionary search control. The two-level policy design is reasonable, but the underlying technical challenges and methodological innovations appear limited. It is unclear what fundamentally new algorithmic insight is introduced beyond integrating these existing components.
The paper does not clearly demonstrate whether the learned policies generalize to unseen materials-design tasks. Since the Workflow Policy is trained using logged outcomes from a small set of tasks, it is unclear whether the model learns transferable decision strategies or task-specific workflow preferences. Moreover, the learned Evolution Policy shows only marginal utility improvements over the rule-based controller on two of the three evaluated tasks (Table 10), and the comparisons use only a single random seed. These limitations weaken the evidence supporting the effectiveness of the learned policies.
Questions for authors
How are the training and evaluation tasks separated for the Workflow Policy? Can the authors provide task-disjoint experiments, such as leave-one-task-out evaluation, to demonstrate generalization to unseen materials-design tasks?
What is the main technical novelty compared with existing experience-learning, adaptive retrieval, and evolutionary search methods? Why are learned LLM-based policies necessary instead of simpler rule-based or lightweight classification approaches?
In Table 10, the utility improvements of the learned Evolution Policy are very small for two tasks. Can the authors provide results across multiple random seeds to establish whether these improvements are statistically reliable?
Since E2A introduces additional policy inference and knowledge retrieval, what is the computational overhead compared with the baselines? Would the improvements remain under a matched total computational budget?
4 comments
Response to Reviewer iTDo (2/2)
2. Generalization and Reliability of the Learned Policies (W2, Q1, Q3)
Workflow Policy generalization (Q1). In the original evaluation (Section 4.3), the Workflow Policy learns from historical returns across the same fourteen-task collection used for evaluation. This tests whether task experience can inform workflow selection, but does not establish generalization to entirely unseen tasks.
To address this concern, we conducted a fourteen-fold task-return-held-out evaluation. In each fold, the target task's returns are excluded from teacher fitting, class weighting, and policy training. The policy then selects a workflow based on the task specification, and this choice is evaluated using frozen historical returns rather than a new online search.
The learned router outperforms fixed M3 on seven held-out tasks, matches it on three, and underperforms on four, including a substantial loss on solid-state electrolytes. These results indicate that workflow preferences can transfer beneficially to some tasks without their historical return labels, while the effectiveness of such transfer remains task-dependent.
Evolution Policy reliability (Q3). The reviewer raises an important question about whether the small improvements reported in Table 10 persist beyond the original random seed. We therefore added two matched seeds for each of the three controller tasks, keeping the full-evidence workflow and search budget fixed.
| Task | Qwen − Rule Discovery Utility (Mean ± SD) |
|---|---|
| Photovoltaic | +0.01162 ± 0.02562 |
| Solid-state Electrolytes | +0.14016 ± 0.03815 |
| Structural Materials | +0.00648 ± 0.00514 |
Paired differences over two additional seeds per task, with 300 search rounds per run; SD denotes sample standard deviation.
Five of the six additional paired runs favor the learned controller, with positive mean improvements across all three tasks. The gains are particularly pronounced on solid-state electrolytes, while Photovoltaic and Structural materials also show improvements. These results further support the effectiveness and reproducibility of learned action selection beyond the original single-seed evaluation.
3. Computational Overhead and Budget Matching (Q4)
Computational Cost. The published baselines include conventional crystal generators such as CDVAE, G-SchNet, DiffCSP, and MatterGen, whose computational costs cannot be directly compared using the token-based accounting applicable to LLM-based generation. We therefore focus on the direct comparison with Independent GPT-4o, using the same generation backbone and 300 search rounds per task.
E2A introduces additional inference costs primarily through richer input contexts, with 95.05% of the additional recorded tokens coming from input. Under a common reference-price scenario, the estimated GPT-4o inference expenditure across all 14 tasks is $58.98 for E2A versus $41.06 for Independent GPT-4o. This corresponds to an additional $17.92 in total, or approximately $1.28 per task on average. Meanwhile, the aggregate observed runtime remains comparable (75,784 s for E2A versus 78,418 s for independent generation).
These results show that E2A achieves the reported improvements in materials-discovery performance with a modest additional GPT-4o inference expenditure per task, without increasing observed task-run time.
A Gentle Follow-up on Further Feedback
Dear Reviewer,
Thank you very much for your time and valuable feedback.
As the discussion period is approaching its end, with fewer than three days remaining, we would kindly like to ask whether you have any further questions or concerns regarding our responses. We would be more than happy to provide additional clarification if needed.
We sincerely appreciate your time, consideration, and constructive feedback.
Best regards, The Authors
Thanks for the detailed response and the additional experiments.
The comparisons with simpler controllers and the additional matched-seed results are helpful. They address some of my concerns about the evolution policy, although the improvements are still relatively small on some tasks.
I also appreciate the task-return-held-out evaluation. The results are mixed though, as the learned workflow policy performs worse than fixed M3 on four tasks. Also, since the evaluation uses historical returns rather than new search runs, I'm still not fully convinced about its generalization to unseen tasks.
As for the novelty, I understand that the main contribution is the overall framework rather than the individual algorithms. Still, I think the technical novelty is somewhat limited, since most of the components are based on existing techniques.
The computational cost analysis is useful too, but using the same number of search rounds doesn't necessarily mean the computational costs are the same.
All in all, some of my concerns have been addressed, but I still have reservations about the technical novelty and generalization.
Response to Reviewer iTDo (1/2)
We thank the reviewer for the thoughtful assessment and constructive questions.
1. Methodological Novelty and Learned Decision-Making (W1, Q2)
We agree that E2A builds on established techniques, including experience feedback, retrieval, offline learning, and evolutionary search. The central methodological question we address, however, is not simply how to incorporate these components, but how accumulated evidence should guide decisions that change the subsequent search process.
Experience feedback and retrieval can provide useful information to a generator, but selecting the appropriate evidence for a task and deciding how to respond to changing search conditions are distinct problems. E2A makes these decisions explicit at two timescales: the Workflow Policy selects an evidence-use configuration before each run, while the Evolution Policy uses the current search state and evidence to select executable EXPLORE, REFINE, or ESCAPE interventions during search. The latter affects candidate selection and search targets, rather than merely adding more text to the generation prompt.
The importance of this distinction is also reflected in our controlled comparisons. With learned action selection disabled, adding Dynamic Knowledge improves discovery on two of the three evaluated tasks, showing that retrieved evidence can contribute independently of the learned controller. Conversely, when the full-evidence workflow is fixed, trained Qwen outperforms both the rule-based controller and unfinetuned Qwen on all three tasks, showing an additional contribution from learned action selection.
We also tested whether an LLM-based controller is necessary by comparing it with rule-based and lightweight alternatives under the same full-evidence workflow and 300-round search budget.
Discovery Utility from matched 300-round active-search comparisons with one seed. Logistic regression and GBDT use an 18-feature numerical representation, while Qwen uses structured textual evidence.
Trained Qwen achieves the highest Discovery Utility on Photovoltaic and solid-state electrolytes, while lightweight classifiers perform better on Structural materials. These comparisons demonstrate that the effectiveness of learned search-action selection varies across tasks and policy models.
The contribution of E2A is therefore the evidence-to-action formulation: separating task-level evidence selection from state-level search intervention, making both decisions learnable, and connecting their outputs to executable changes in a closed-loop discovery process. The individual retrieval and evolutionary operators are established techniques; the methodological contribution lies in how their use is decided and evaluated.