acceptodds
Under review as a conference paper at ICLR 2027

Divergent Evolution Paths for Multi-Objective Molecular Optimization

Abstract

Multi-objective molecular optimization is fundamentally constrained not only by the vastness of chemical space, but by the intrinsic conflict among objectives: modifications that improve one property often degrade others. Existing approaches typically optimize molecules within a single evolving search context, which entangles conflicting objectives and discards the path-specific experience needed to reach distinct Pareto regions. We propose ATOM (Agents on a Tree for multi-Objective Molecular optimization), a trajectory-centric framework that treats alternative optimization histories, rather than individual molecular candidates alone, as the units of Pareto exploration. Instead of enforcing global consensus, ATOM organizes specialized agents along different branches of a tree-structured search, where each agent evolves molecules toward a particular objective while accumulating trajectory-specific structural experience. To support structured knowledge sharing, ATOM incorporates three complementary coordination mechanisms operating at lateral, hierarchical, and global scales, enabling progressive cross-path optimization. Experiments on molecular optimization benchmarks involving predicted bioactivity, synthetic accessibility, and CYP3A4 non-inhibition show that ATOM attains the best mean aggregate hypervolume and competitive Pareto coverage, with gains more pronounced under severe inter-objective conflict. These results suggest that divergent evolution provides an effective paradigm for complex multi-objective molecular design. Code is available at https://anonymous.4open.science/r/ATOM-41CE.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM, Agents, Molecular property optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Zz1fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ATOM, a framework for multi-objective molecular optimization that combines property-specific LLM agents with Monte Carlo tree search. The method explores different sequences of molecular optimization steps while keeping track of each search path's history. Agents exchange useful molecules and strategies through local coordination and a shared Pareto memory. Across eight tasks, ATOM achieves the best mean hypervolume on six. Ablation studies suggest that knowledge sharing and maintaining consistent path histories both contribute to performance.

Critical strengths
  1. I find the use of optimization history interesting. ATOM allows different search branches to follow their own sequences of optimization steps while still sharing useful information.

  2. The ablations are well designed. Together, the flat-coordination and path-history controls help test whether the gains come from tree structure and consistent path histories, rather than just providing more agents or additional context.

Critical weaknesses
  1. The linked implementation differs substantially from the method described in the paper. Equations 5–8 use local hypervolume rewards and UCT, whereas the available code uses mean property scores and different selection and backup rules. If this is a versioning issue, the authors could clarify it.

  2. It is unclear how path history is represented and updated in practice. Sections 3.1 and 4 describe ordered specialist histories and accumulated optimization strategies, while Appendix E describes the path strategy as a short summary of objective status and the current best candidate. How does the structural-edit summarizer in Figure 11 relate to this representation? A concrete example showing how the history changes along a branch would be helpful.

Questions for authors

As mentioned in the Weaknesses, could the authors clarify the discrepancies between the released code and the equations in the paper? Also, how exactly is path history represented and updated in practice?

4 comments

Dear Reviewer Zz1f,

Thank you for identifying where the release and manuscript need to be aligned. We clarify the search reward and the implementation of path history below.

Q1. The linked implementation differs substantially from the method described in the paper. Equations 5–8 use local hypervolume rewards and UCT, whereas the available code uses mean property scores and different selection and backup rules. If this is a versioning issue, the authors could clarify it.

Response. The anonymous repository contained an earlier version of ATOM that guided search using the mean of normalized objective scores. All results in Table 1 were generated with the manuscript implementation, which uses local hypervolume rewards and the UCT selection and backup rules described in Equations 5–8. The repository mismatch therefore does not affect the reported results or the paper’s conclusions.

Q2. How is path history represented and updated?

Response. In the experimental implementation that uses the strategy summarizer, each child node records the specialist that created it, its molecular population, and a short strategy note. Its population includes candidates retained after scoring and filtering, along with selected references from its parent’s Pareto front. The parent links recover the order of specialist calls along the branch.

When a specialist expands a node, it uses the node’s population, retrieved references, and strategy note to propose edits. After filtering, the Figure 11 summarizer compares the parent population with the newly retained candidates as two unpaired sets. It records up to five recurring structural edits in the child’s note. If it finds no clear pattern, the child inherits the parent’s note.

For example, after a QED specialist expands a node, the summarizer may identify ring formation as a recurring edit. An SA specialist can then use that note and the QED node’s molecular references when proposing its own edits. If amide formation emerges as a recurring edit among the retained SA candidates, the SA child stores it in its own note for later expansions.

Appendix E describes a deterministic summary of objective status and the current best candidate. In the configuration described above, the strategy passed to later specialists is the structural-edit note from Figure 11.

We appreciate your insightful feedback and suggestions, which have helped us strengthen the presentation, validation, and overall quality of ATOM.

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly. We sincerely look forward to your feedback.

Thank you for your response. The given case helps clarify how path history is represented and used in ATOM. However, some inconsistencies remain between the paper and the released code, including the implementation differences I raised and the issue noted by Reviewer 6sLB. Given these remaining concerns, I will maintain my original score.

Thank you for following up and for pointing out these inconsistencies. The manuscript PDF and the anonymous code repository had not yet been updated when we posted our earlier response. Both have now been updated to address the discrepancies you and Reviewer 6sLB identified. The changes are marked with blue boxes in the revised PDF. We appreciate your careful review and feedback.

Reviewer_6sLBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ATOM, a multi-agent framework for multi-objective molecular optimization. It combines property-specific LLM experts, tree search over optimization histories, and knowledge exchange across branches. Experiments evaluate Pareto hypervolume across eight molecular optimization tasks.

Critical strengths
  1. The framework provides an explicit way to explore different sequences of property optimization while preserving their histories.

  2. The flat-coordination and history-disruption ablations provide useful evidence for the proposed design (Tables 2–3).

  3. The evaluation includes multiple objective combinations, five random seeds, and an additional LLM backbone.

Critical weaknesses
  1. Although the ablations support several design choices, improvements over MoLLEO are uneven across tasks. The added value of the more complex search and coordination framework remains insufficiently established (Table 1).

  2. Correlations measured on random ZINC20 molecules do not adequately establish optimization conflict or support the claim of greater benefits under stronger conflict. Evaluation should include medicinally motivated property trade-offs whose conflict is explicitly demonstrated (Appendix D).

  3. Important related work is missing, including MolSearch [1], Mothra [2], PMMG [3], and InversionGNN [4]. Their discussion would help clarify the contribution beyond existing multi-objective search methods.

  4. The released code appears inconsistent with the paper. For example, compute_property in ATOM/utils.py returns the raw CYP3A4 inhibition score rather than the stated non-inhibition objective (Appendix B, Eq. 14); the implementation used for the reported experiments needs clarification. This one should be verified before acception.

References

[1] Mengying Sun et al. MolSearch: Search-based Multi-objective Molecular Generation and Property Optimization. KDD, 2022.

[2] Takamasa Suzuki et al. Mothra: Multiobjective de novo Molecular Generation Using Monte Carlo Tree Search. Journal of Chemical Information and Modeling, 2024.

[3] Yifei Liu et al. A Multi-Objective Molecular Generation Method Based on Pareto Algorithm and Monte Carlo Tree Search. Advanced Science, 2025.

[4] Yifan Niu et al. InversionGNN: A Dual Path Network for Multi-Property Molecular Optimization. ICLR, 2025.

Questions for authors

See above.

4 comments

Dear Reviewer 6sLB,

Thank you for your constructive feedback. We address the comparison with MoLLEO, the evidence for objective conflict, the related work, and the CYP3A4 objective below.

Q1. Although the ablations support several design choices, improvements over MoLLEO are uneven across tasks. The added value of the more complex search and coordination framework remains insufficiently established (Table 1).

Response. We further expanded the evaluation to test whether ATOM’s tree search and coordination add value when MoLLEO is retained as the molecular optimizer. As shown in Table R3, replacing each specialist with a complete single-objective MoLLEO optimizer and using ATOM to guide tree search and coordinate molecules improves final HV over standalone MoLLEO on all three tasks. The main conclusion remains unchanged: ATOM’s tree search and coordination provide consistent gains beyond the molecular optimizer itself. For example, ATOM–MoLLEO improves HV from 0.694 to 0.799 on JNK3 + GSK3β, with the same trend on QED + SA (0.927→0.942) and JNK3 + GSK3β + CYP3A4 inhibition (0.643→0.701). These results show that retaining MoLLEO as the molecular optimizer does not remove the benefit of ATOM’s tree search and coordination.

TaskMoLLEOATOM–MoLLEOΔHV
QED + SA0.927 ± 0.0040.942 ± 0.001+0.015
JNK3 + GSK3β0.694 ± 0.0100.799 ± 0.030+0.105
JNK3 + GSK3β + CYP3A4 inhibition0.643 ± 0.0310.701 ± 0.010+0.058

Table R3. Final HV for standalone MoLLEO and the ATOM–MoLLEO hybrid. ATOM–MoLLEO uses MoLLEO as the node optimizer. Values are three-seed mean ± standard deviation; ΔHV = hybrid mean − standalone mean.

Q2. Correlations measured on random ZINC20 molecules do not adequately establish optimization conflict or support the claim of greater benefits under stronger conflict. Evaluation should include medicinally motivated property trade-offs whose conflict is explicitly demonstrated (Appendix D).

Response. Objective conflict has been formally studied in multiobjective optimization (Brockhoff and Zitzler, 2009). In medicinal chemistry, matched molecular pair analysis has been used to examine how local structural changes affect multiple pharmaceutical properties (Leach et al., 2006).We measured objective conflict directly in ATOM’s search trajectories. A proposal was counted as conflicting if, relative to the mean scores of its parent population, one normalized objective improved by more than 0.05 while another declined by more than 0.05. We evaluated drug-likeness (QED), estimated ease of synthesis (SA), and predicted inhibition of JNK3 and GSK3β. In the four-objective task, 58.2% of proposals that improved both predicted inhibition scores by more than 0.05 also reduced QED or normalized SA by more than 0.05.

As shown in Table R4a, across the three tasks, the observed conflict rate increased from 6.7% to 33.2%, and ATOM’s hypervolume (HV) advantage over the single-agent baseline increased from 0.061 to 0.173. This pattern is consistent with larger gains on tasks with more frequent conflict.

TaskConflict rateSingle-agent HVATOM HVΔHV
QED + SA6.7%0.8800.941+0.061
JNK3 + GSK3β13.6%0.6210.790+0.169
QED + SA + JNK3 + GSK3β33.2%0.4330.606+0.173

*Table R4a. Conflict rates along ATOM’s search trajectories and mean HV from Table 1. *

We also evaluated two medicinally motivated lead optimization tasks: increasing predicted JNK3 inhibition while reducing predicted hERG blockade, a cardiac safety concern, or reducing predicted JNK2 inhibition to improve selectivity. Both trade-offs were observed experimentally in a JNK3 inhibitor series (Feng et al., 2021). As shown in Table R4b, ATOM achieved the highest HV on both tasks, exceeding the strongest baseline by 0.052 on the hERG task and 0.013 on the JNK2 task.

TaskATOMMAMOMoLLEOGraph GAGP BO
JNK3 inhibition + reduced hERG blockade0.8030.7510.7220.6110.578
JNK3 inhibition + reduced JNK2 inhibition0.6430.5870.6150.6300.460

Table R4b. Final HV on two lead optimization tasks motivated by experimentally observed trade-offs.

Q3. How does ATOM differ from MolSearch, Mothra, PMMG, and InversionGNN?

Response. Table R5 compares the search units and optimization strategies.

MethodSearch representationOptimization strategy
MolSearchIndividual molecular structuresRule-based transformations with two-stage MCTS
MothraPartial SMILESRNN-guided Pareto MCTS
PMMGPartial SMILESRNN-guided search with vector-valued UCB
InversionGNNMolecular graphsGNN-guided gradient-based Pareto optimization
ATOMMolecular populations with branch historiesProperty-specialized LLM agents with coordination across tree branches

Table R5. Comparison with existing multi-objective molecular optimization methods.

ATOM makes three contributions that complement the search strategies in Table R5.

  1. Population-level search with branch history. Each node holds a population of complete molecules. Parent links and strategy notes record how each branch evolved, so different branches can explore different sequences of property-directed edits.
  2. Property specialists using a shared LLM. Each specialist proposes structural edits for one objective using the branch’s reference molecules and strategy note. All specialists use the same frozen LLM.
  3. Coordination across branches. Child nodes inherit molecular references and edit guidance from their parents, sibling branches exchange selected molecules, and a global Pareto archive makes promising trade-offs available throughout the tree.

Together, these mechanisms let ATOM develop distinct optimization paths and reuse promising candidates across them. We will clarify these contributions in the revised manuscript.

Q4. Which CYP3A4 objective was used in the reported experiments?

Response. The original reported CYP3A4 experiments maximized the predicted inhibition score returned by ATOM/utils.py. Appendix B, Eq. (14) instead described the complement, 1−CYP3A41-\mathrm{CYP3A4}, which was inadvertently introduced during manuscript revision. The score measures predicted inhibitory propensity toward a key drug-metabolizing enzyme and provides a target-specific objective for studying trade-offs with other molecular properties. We will correct Eq. (14) and identify the objective as predicted CYP3A4 inhibition wherever we describe the original benchmark results.

The corrected equation will match the objective already used to compute the reported HV values and method comparisons. Those results and the component ablations continue to support our conclusions about multi-objective optimization and our core contributions in tree-structured search and coordination among property specialists.

Thank you for helping us clarify the experimental protocols and the evidence for our design choices and respectfully hope that the strengthened manuscript merits positive consideration.

References

  • Brockhoff, D., and Zitzler, E. (2009). Objective reduction in evolutionary multiobjective optimization: Theory and applications. Evolutionary Computation, 17(2), 135–166.
  • Feng, Y., Park, H., Ryu, J. C., and Yoon, S. O. (2021). N-Aromatic-Substituted Indazole Derivatives as Brain-Penetrant and Orally Bioavailable JNK3 Inhibitors. ACS Medicinal Chemistry Letters, 12(10), 1546–1552.
  • Leach, A. G., Jones, H. D., Cosgrove, D. A., Kenny, P. W., Ruston, L., MacFaul, P., Wood, J. M., Colclough, N., and Law, B. (2006). Matched molecular pairs as a guide in the optimization of pharmaceutical properties: A study of aqueous solubility, plasma protein binding and oral exposure. Journal of Medicinal Chemistry, 49(23), 6672–6682.
  • Sanguinetti, M. C., Jiang, C., Curran, M. E., and Keating, M. T. (1995). A mechanistic link between an inherited and an acquired cardiac arrhythmia: HERG encodes the IKr potassium channel. Cell, 81(2), 299–307.
  • Shuai, W., Bu, F., Zhu, Y., Wu, Y., Xiao, H., Pan, X., Zhang, J., Sun, Q., Wang, G., and Ouyang, L. (2023). Discovery of Novel Indazole Chemotypes as Isoform-Selective JNK3 Inhibitors for the Treatment of Parkinson’s Disease. Journal of Medicinal Chemistry, 66(2), 1273–1300.

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly. We sincerely look forward to your feedback.

Thank you again for your careful review. As the discussion period closes this evening, we wanted to follow up briefly. We corrected Eq. (14) to reflect the CYP3A4 inhibition score used in the reported experiments and added a MoLLEO comparison and new analyses of objective conflict.

If you have a chance before the deadline, we would welcome any remaining questions. If the additional evidence and clarifications address your main concerns, we would be grateful if you would consider updating your rating in light of the rebuttal.

Reviewer_wymaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A multi-objective molecular optimization setup that utilizes a tree of specialized agents. Experiments are based on standard molecular optimization benchmarks for various bioactivity metrics and general molecular metrics.

Critical strengths

The methodology is quite interesting as the setup with various agent experts for a specific molecular objective is intuitive and simple to understand. The experiments appear quite strong over the tasks tested. There are some interesting ablations and case studies.

Critical weaknesses

The metrics tested are all quite simple QED, SA, GSK3B, JNK3 are all relatively simple objectives. It would be nice to see the results on more complex objectives. The computational resources needed appear to be much higher with the need of training for various experts for the various properties. The scalability when the objective is much higher seems questionable. The performance at low oracle calls appears really weak.

Questions for authors

The performance at low oracle calls is really weak. Is the cause from utilizing the tree structure with various experts? Or is there some other explanation?

How does the method perform on more difficult molecular tasks?

Can information on the computational resources be included in the main text?

How is the scalability to a higher number of objectives?

5 comments

Dear Reviewer wyma,

Thank you for raising questions about ATOM’s evaluation budgets, computational cost, and performance on more demanding tasks. We address each point below.

W1. Training and computational cost of the specialist agents

Response. ATOM performs molecular optimization at inference time. All specialists use the same frozen, general-purpose GPT-4o-mini model through an API. Objective-specific prompts and molecular references define their roles. During a run, ATOM updates the search tree, molecular populations, Pareto memory, and strategy summaries while keeping the LLM weights and property oracles fixed. We will clarify in the main text that the specialist roles require no task-specific training or fine-tuning.

Q1.The performance at low oracle calls is really weak. Is the cause from utilizing the tree structure with various experts? Or is there some other explanation?

Response. The ablation results suggest that both design choices contribute to ATOM’s weaker performance at the tightest budget. At 50 oracle calls, replacing tree search with flat search while keeping the specialists raised mean ΔHV from 0.0032 to 0.0077. Keeping tree search and replacing the specialists with a single generalist raised it to 0.0129 (Table R5).

Flat search with specialists still led at 100 and 200 calls, while full ATOM overtook tree search with a generalist by 200 calls. At 300 calls, full ATOM achieved the highest mean ΔHV (0.0559), slightly ahead of flat search with specialists (0.0534). This pattern suggests that both components can slow early progress. However, as shown by the cumulative-HV curves in Figure 3, this early-budget effect is much less pronounced in the two- and three-objective tasks, and practical molecular optimization campaigns rarely operate at such small oracle budgets; therefore, this phenomenon should have limited impact on typical molecular optimization tasks.

SettingΔHV@50ΔHV@100ΔHV@200ΔHV@300
Full ATOM0.00320.01390.02220.0559
Flat search + multiple specialists0.00770.02250.03110.0534
Tree search + generalist0.01290.01370.01460.0157

Table R5. Ablation results at four oracle budgets. Bold indicates the highest mean at each budget.

Q2. How does ATOM perform on more difficult molecular tasks?

Response. We evaluated ATOM on a four-objective CNS design task that combines predicted JNK3 and GSK3β inhibition with blood–brain barrier penetration and hERG non-blockade. We used ADMET-AI to score the two ADMET objectives. We then added synthetic accessibility (SA) for the five-objective task and drug-likeness (QED) for the six-objective task.

TaskATOMMoLLEOMAMO
JNK3 + GSK3β + BBB + hERG non-blockade0.4130.3930.270
JNK3 + GSK3β + BBB + hERG non-blockade + SA0.3200.3290.244
JNK3 + GSK3β + BBB + hERG non-blockade + QED + SA0.2880.1900.171

Table R2. HV at a common oracle budget within each task.

These results show that ATOM remains competitive when predicted kinase activity is optimized jointly with ADMET and molecular design properties. ATOM achieved the highest HV among the three methods on the four- and six-objective tasks. On the five-objective task, it outperformed MAMO and came within 0.010 HV of MoLLEO at the same budget.

Q3. Can the computational resources be reported in the main text?

Response. Under the main benchmark’s 10,000-evaluation budget, ATOM averaged 2 h 23 min and 2,691 external LLM generation requests per run over five seeds. The reported setup used an Intel Xeon CPU and an NVIDIA A800 GPU for local computation and accessed GPT-4o-mini through an external API.

We will move these figures and the hardware configuration from Appendix Table 11 to the main text so readers can assess the cost of running ATOM.

Q4. How does ATOM scale to more objectives?

Response. Adding an objective requires a property scorer and an additional specialist prompt. The specialists continue to share the same frozen LLM and search framework. The results in Table R2 show that this same framework remains competitive with MoLLEO and MAMO up to six simultaneous objectives in the settings we evaluated.

We appreciate your suggestions on how to make the budget and resource comparisons clearer and respectfully hope that the strengthened manuscript merits positive consideration.

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly. We sincerely look forward to your feedback.

Thank you for your response. I have no further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.