- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes OS-DEV, a training-free agentic framework for cold-start molecular optimization. OS-DEV utilizes an LLM to propose candidates, and instantiates a pair of analysts to distill skills from feedback for the candidates. Then, a verifier evaluates the evidence for the skills and assigns a confidence label for each skill. The skills then guide the LLM optimizer towards better candidates. Experimental results on DOCKSTRING show that OS-DEV outperforms representative baselines, and ablation studies demonstate that each component of OS-DEV brings a performance gain.
Critical strengths
-
This paper is easy to follow. The motivation is clearly stated, and the method details and experiments are explained well.
-
The proposed OS-DEV does not reply on pre-built skill libraries, and does not introduce new trainable parameters, which makes it easy to deploy.
-
It is a reasonable idea to instantiate a pair of analysts for analyze successes and failures respectively.
-
OS-DEV outperforms representative baselines.
Critical weaknesses
-
The quality and reliability of the distilled skills are not directly verified. Although the ablation studies show the performance gain brought by each component, there is no experimental results to support that the distilled skills are reliable, or that the verifier can indeed tell good rules from misleading ones. In addition, whether high-confidence skills are more useful than low-confidence ones is also not validated, and so it is hard to say that the confidence labels work as expected.
-
The advantage of the proposed components (e.g., the pair of success and failure analysts) over existing techniques with similar functionality is not verified. For example, ACE's reflector extracts experience from both successes and errors, how does such a unified reflector compare with the proposed analyst pair?
-
The performance gains from some components are modest (e.g. failure analyst).
Questions for authors
See "weaknesses". One more: as stated in Sec.2.1, the oracle returns a scalar per candidate and nothing else, while the analyst prompt includes QED/vina components (Sec.3.2). Could the authors clarify this?
3 comments
Rebuttal to Reviewer u55M [2/3]
W2. Unified reflector versus two analysts
Thank you for this thoughtful suggestion. A unified reflector can consider successful and failed trajectories together and draw on their contrast; separate analysts instead let each prompt focus on a narrower task: the success analyst identifies promising patterns, while the failure analyst concentrates on degradation and invalid proposals (Section 3.2, lines 225–242). Both are reasonable designs. Trace2Skill similarly separates success and error analysis, although its tasks and consolidation pipeline differ from ours.
We tested the reviewer’s alternative by giving both successful and failed trajectories to a single analyst, keeping the rest of OS-DEV unchanged. On seven targets with three seeds, the unified reflector reaches 0.9004 and the dual analysts reach 0.9019:
| Analyst design | Mean score (7 targets, 3 seeds) |
|---|---|
| Unified reflector | 0.9004 |
| Dual analysts (OS-DEV) | 0.9019 |
The unified design is a strong alternative: the dual design is higher by a modest 0.0015 in this experiment. We appreciate the suggestion, which helps clarify that specialization is useful here but joint reflection is also a viable choice.
Rebuttal to Reviewer u55M [3/3]
W3. Is the failure analyst worth its modest gain?
We agree that the success analyst contributes more in this setting: on 58 targets, success-only reaches 0.8426, while adding the failure analyst yields 0.8487 (+0.0061; Section 4.3, lines 424–427). The failure analyst contributes a complementary signal by examining degraded candidates and invalid SMILES to identify patterns to avoid. With a limited budget of expensive molecular evaluations, we believe this additional guidance is worthwhile.
The two analyst calls run in parallel, limiting the additional elapsed time when concurrent inference is available. Neither analyst uses additional oracle evaluations.
Question. Scalar objective versus QED/Vina in the trace
Thank you for pointing this out; we apologize that our description was unclear. The oracle returns the scalar objective in Eq. (1). For the analyst trace, we calculate QED from the candidate SMILES using RDKit, and recover the Vina component from and QED using the objective formula. This does not require an additional docking evaluation. We clarify this data flow in Section 2.1, lines 145–151, and the Implementation and Computational Cost appendix.
Specifically, with the normalization in Eq. (1), the normalized Vina component is
and the raw Vina score is
The reconstruction assumes sufficient numerical precision.
Rebuttal to Reviewer u55M [1/3]
Dear Reviewer, thank you for the encouraging assessment and concrete suggestions. We have added controls for the verifier and analyst design and clarify the oracle information below.
W1. Can the verifier distinguish useful skills?
Thank you for this suggestion. Owing to the limited time and compute available during the discussion period, we ran two complementary tests on seven targets (the first listed target from each protein family) with three seeds each; we apologize for the limited coverage. We will continue these experiments and include the results in the paper once completed.
First, without calling the verifier, we independently assign each skill a high, medium, or low label with equal probability. This random-label variant reaches 0.8487, versus 0.9019 for OS-DEV:
Second, we run the verifier and retain different groups of skills; the optimizer still sees their original confidence labels. The results are:
High + medium reaches 0.8992, compared with 0.8664 for low + medium; high only reaches 0.8915, compared with 0.8521 for low only. These results show that higher-confidence skills provide more useful guidance for optimization, supporting the effectiveness of the evidence-aware verifier.