- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents CoSyn, a literature-grounded framework for pharmaceutical co-crystal synthesis design. The authors construct CoSyn-DB, containing 20,783 structured synthesis records extracted from scientific literature through a hierarchical, method-aware extraction pipeline. The database is subsequently used for method-specific synthesis-condition prediction with retrieval-augmented few-shot LLMs. Finally, the authors integrate literature analysis, database retrieval, and condition prediction into single-agent and multi-agent systems.
The proposed framework is evaluated through manually annotated extraction datasets, comparisons with classical machine-learning baselines, prospective synthesis experiments on four molecular pairs, and a newly constructed 30-question benchmark (CoSyn-Bench). The results demonstrate the feasibility of literature-grounded condition recommendation and improved benchmark performance through tool-augmented workflows.
Critical strengths
S1. A useful structured resource for an underexplored downstream task.
The construction of CoSyn-DB, comprising 20,783 method-specific synthesis records, addresses the practical challenge of organizing heterogeneous experimental information from the literature. Unlike conventional co-crystal screening tasks, the proposed resource connects molecular pairs with synthesis conditions, potentially enabling further research on synthesis-condition recommendation.
S2. Experimental validation beyond purely computational evaluation.
The paper evaluates extraction and prediction components independently and includes prospective experimental validation. Obtaining phase-pure co-crystals for three of four tested molecular pairs provides preliminary evidence that literature-grounded recommendations can support practical synthesis experiments. The inclusion of tool ablations and independent expert evaluation also strengthens the empirical assessment.
Critical weaknesses
W1. The necessity and distinctive contribution of LLM-based prediction and agentic orchestration remain insufficiently established.
The strongest contribution appears to be the construction of a structured synthesis database rather than a novel LLM-based methodology. In Table 3, the LLM predictors achieve an Overall F1 comparable to the Random Forest baseline, making their additional predictive value unclear. Furthermore, although the agentic systems outperform LLM-only and static-context baselines, the experiments do not fully distinguish the benefits of agentic orchestration from those of accessing domain-specific tools and structured evidence.
W2. The system-level evaluation has limited scope and generalizability.
CoSyn-Bench contains only 30 questions designed around the capabilities of the proposed system. Some tasks primarily test database lookup or predefined combinations of retrieval and prediction, raising concerns about whether the benchmark captures broader synthesis-design reasoning. Although LLM-as-a-judge evaluation is supported by human agreement analysis, the expert evaluation covers only the two full agentic architectures, leaving the reported improvements over non-agentic baselines without direct human validation.
W3. The evidence for general synthesis-design capability remains preliminary.
The prospective evaluation is encouraging but includes only four literature-confirmed molecular pairs under a fixed synthesis method. This demonstrates the potential usefulness of predicted starting conditions, but does not establish generalization to broader synthesis methods, previously unconfirmed molecular pairs, or more challenging experimental settings. The relatively limited record-level exact-match performance also raises questions about the reliability of downstream recommendations for sensitive experimental parameters.
Questions for authors
Q1. Necessity of LLM-based condition prediction.
In Table 3, the LLM predictors achieve Overall F1 scores comparable to the Random Forest baseline, despite using additional retrieved literature examples. This raises concerns about whether the LLM component provides a meaningful predictive advantage over conventional methods, particularly given its additional computational cost and complexity. Since LLM-based condition prediction is a central component of the proposed framework, the lack of a clear performance improvement makes its methodological contribution difficult to assess. How does CoSyn-Predictor compare against a simple retrieval-only baseline that directly transfers or aggregates synthesis conditions from the nearest literature examples?
Q2. Contribution of agentic orchestration.
Although the tool ablations demonstrate the importance of individual capabilities, the comparisons do not fully isolate the benefit of agentic orchestration. How would a deterministic, non-agentic workflow with access to the same retrieval, analysis, and prediction tools perform under a comparable execution budget?
Q3. Generalizability beyond the self-constructed benchmark.
Since CoSyn-Bench is designed around the predefined capabilities of the proposed framework, strong performance may primarily reflect successful execution of these specific workflows rather than generalizable scientific reasoning or synthesis-design ability. Have the authors evaluated the system on an independent, more general, or out-of-distribution benchmark involving synthesis-design tasks beyond those explicitly covered by CoSyn's tools?
Q4. Novelty of synthesis recommendations.
The prospective experiments validate starting conditions for four literature-confirmed co-crystal-forming pairs under a fixed synthesis method. However, it remains unclear whether the framework can scientifically reason about and propose novel synthesis conditions beyond transferring or adapting existing literature precedents. Can the authors provide experimental evidence or concrete examples where CoSyn successfully proposes and validates novel synthesis conditions that are not directly supported by closely related literature records?
5 comments
Author Response [Part 2/2]
[W2/Q3]. Benchmark validity
CoSyn-Bench was deliberately designed to evaluate literature-grounded co-crystal synthesis assistance. To our knowledge, no established external benchmark directly assesses method-specific synthesis-condition recommendation and the coordination of supporting tools. We therefore developed 30 expert-curated single- and multi-capability questions to evaluate these workflows in a consistent setting. We have clarified this intended scope in Section 5.1.
To strengthen the reliability of this evaluation, we expanded the independent expert assessment to 120 responses from four systems, evaluated by two experts. The LLM-expert Spearman correlation (0.806) was comparable to inter-expert agreement (0.792). Beyond measuring agreement, we also compared expert-assigned scores across systems, with MA achieving the highest mean Overall Score of 3.63 (Appendix C.7, Tables 32-33).
The expanded expert assessment strengthens confidence in the comparisons made within CoSyn-Bench. Assessing broader applicability would require testing CoSyn across a wider range of co-crystal syntheses, including more challenging molecular systems, and evaluating its use in experimental workflows.
[W3/Q4]. Experimental applicability
Adapting established synthesis knowledge to specific co-crystal pairs is precisely the task CoSyn addresses. CoSyn-DB organizes a broad range of experimentally reported conditions, providing a foundation for selecting suitable quantities, solvents, and processing parameters for a target pair, even when its own synthesis protocol is unavailable.
We tested this capability prospectively, excluding all records for four target pairs before generating recommendations. The proposed conditions were pair-specific: CoSyn recommended an acetonitrile–water mixture with 12 h stirring for caffeine–oxalic acid, and water with 48 h stirring for urea–oxalic acid. Both yielded phase-pure co-crystals (Appendix B.13). Overall, CoSyn produced three phase-pure co-crystals in four experiments, while the baseline yielded two mixed products and two failures (Table 4). As clarified in Section 4.3, unavailable ethyl acetate was replaced with 1,2-dimethoxyethane based on similar relative polarity, which limits the comparison.
We also examined extraction reliability. The 25.3% Full-record Exact Match requires agreement across all protocol components, including processing details. Our new hierarchical metrics show higher agreement for chemical composition (65.2%) and quantitative composition (52.5%) (Appendix A.11, Table 12). Our error-propagation analysis also found comparable downstream Overall F1 using manually annotated and extracted records (0.23 compared with 0.24, Appendix B.11, Table 23), indicating no substantial aggregate degradation in the evaluated setting.
These findings support the practical value of CoSyn-DB as a structured evidence resource and demonstrate the initial feasibility of literature-grounded synthesis-condition recommendation. Broader prospective validation across synthesis methods and more challenging molecular systems remains an important next step.
We believe these additions strengthen the evaluation of CoSyn and clarify its contributions to literature-grounded synthesis design. We would be grateful if the reviewer would reconsider their rating in light of the new evidence.
Revision: Table 3 (L277–L285), Section 4 (L297–L315), Sections 4.2–4.3 (L345–L385), Sections 5.1–5.2 and Table 5 (L399–L442), Section 6 (L516–L517), Appendix A.11 (L1694–L1707), Appendix B.12 (L2291–L2314), Appendix B.13.1 (L2324–L2325), Appendix C.3.2 (L2634–L2747), Appendix C.6–C.7 (L3090–L3156).
Thank you for the detailed response and additional experiments. I have two remaining questions.
Q2/Q3. Statistical significance of agentic improvements
Given the small size of CoSyn-Bench (30 questions, including only 10 multi-capability questions), could the authors report 95% confidence intervals for the performance differences between agentic and non-agentic baselines, using paired bootstrap resampling over questions? This would help establish whether the reported improvements from agentic coordination are statistically reliable.
Q4. Novelty of the recommended synthesis conditions
Is it possible to quantify the number of possible and unique synthesis-condition combinations based on the parameter values in CoSyn-DB, and assess how many previously unseen combinations were experimentally validated by CoSyn? For instance, comparing the recommended conditions (e.g., solvents, temperature, time, and stoichiometric ratios) with the top-3 retrieved examples and broader CoSyn-DB records, alongside molecular similarity, could help clarify the added value of LLM-based prediction beyond direct retrieval of existing protocols.
Author Response [Part 1/2]
We thank the reviewer for recognizing the value of CoSyn-DB and our prospective experimental validation. To address the concerns raised, we added controlled baselines, expanded expert evaluation, and clarified the scope and reliability of our synthesis recommendations.
[W1/Q1]. Value of LLM prediction
CoSyn-DB provides the structured experimental evidence needed to study protocol-level synthesis-condition prediction. Our goal is to assess how effectively LLMs use this evidence compared with simpler approaches. We therefore added two retrieval-only baselines: NN-1 directly transfers conditions from the nearest literature record, while kNN-3 aggregates conditions from the three nearest records.
On the held-out grinding set, Gemini-2.5-Flash achieves an Overall-Single F1 of 0.49, compared with 0.46 for NN-1, 0.47 for kNN-3, and 0.46 for Random Forest. Its Numerical F1 is slightly higher than the retrieval-only baselines (0.48 compared with 0.46), while Random Forest performs better on solvent and categorical parameters (Table 3). We also added a molecular-similarity analysis, showing that Gemini-2.5-Flash maintains an Overall F1 of 0.48 in both the lower- and higher-similarity subsets (Appendix B.12).
These results show that LLM-based prediction is competitive but does not consistently outperform simpler methods. The added comparisons provide a clearer understanding of where LLMs may offer benefits and where direct retrieval remains sufficient for literature-grounded synthesis-condition recommendation.
[W1/Q2]. Value of agentic coordination
To isolate the value of agentic coordination beyond tool access, we added two non-agentic baselines using the same GPT-4o-mini backbone, domain tools, and evidence as Single-Agent (SA) and Multi-Agent (MA). Fixed Workflow (FW) executes a predefined tool sequence, while LLM-Routed (LR) selects a predefined workflow without replanning.
The results reveal a clear distinction by task complexity. On multi-capability questions, SA and MA achieve scores of 3.25 and 3.35, compared with 2.15 for FW and 2.63 for LR. On single-capability questions, however, FW and LR remain competitive and slightly outperform the agents (Table 5). This suggests that flexible coordination is most useful when a task requires combining several capabilities, rather than executing a predefined workflow.
We also examined the execution costs of this flexibility. FW and LR require fewer LLM calls, tokens, and less time than the agentic systems. Among the agents, MA uses considerably fewer tokens than SA (20,130 compared with 57,611), but requires more calls and longer execution time (Appendix C.6, Table 31). The added baselines thus reveal a clear quality-efficiency trade-off: non-agentic workflows are more economical for single-capability tasks, while agentic coordination is more beneficial for multi-capability tasks, with SA offering lower latency and MA lower token consumption.