- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduced a framework named ELMS which utilizes Monte Carlo Tree Search along with LLM and evaluator feedback for motif scaffolding. The method is evaluated on GeomMotif and MotifBench under a fixed budget and reports substantially higher task success rates than existing generative baselines.
Critical strengths
- The paper proposed a useful solution for motif scaffolding, using the evaluator feedback as signal rather than using generate-filter process.
- The system is carefully modularized into non-LLM and LLM part. The authors provide implementation details, which improve the clarity and reproducibility of the proposed framework.
Critical weaknesses
- ESMFold is used during optimization and evaluation, expecially on the GeomMotif task. The final criterion is nearly the same as the optimization reward (pLDDT>70 and RMSD<1). This is doing some kind of reward hacking, I wonder a simple protein language model + MCMC can also performs well in the task. From Table 9 we can also find that the Unique score of ELMS is extremely low compared with previous design methods like La-Proteina and RFDiffusion. I suggest that the authors should evaluate the final designs with an independent structure predictor, though these structure prediction models might show similar distribution.
- As I said in 1., I wonder how simple PLM+MCMC might perform under the same setting. I noticed there is greedy search baseline implemented but I still wonder some other standard evaluator-guided search methods under the same evaluation budget.
- The contribution of MCTS is difficult to isolate. The key idea of the paper should be MCTS, while the method introduces many additional components, including adaptive operators and static action arms. How are these components chosen, and how much of the improvement comes from these hand-designed priors rather than MCTS itself?
- The necessity of the LLM agents is unclear. It looks like the Critic Agent is mainly doing parsing work (the problem is actually deterministic according to Table 10) and the Policy Agent selects from operators already ranked by the operator router.
Questions for authors
- I would like to see some simple evaluator-guided method and using seperate protein folding tool in optimization and evaluation.
- Could you help to explain the neccessity of LLM agents and how it will distinguish from deterministic rule?
5 comments
reply2
Q1: I would like to see some simple evaluator-guided method and using seperate protein folding tool in optimization and evaluation. + Weakness: I suggest that the authors should evaluate the final designs with an independent structure predictor, though these structure prediction models might show similar distribution. + Weakness: ESMFold is used during optimization and evaluation, expecially on the GeomMotif task. The final criterion is nearly the same as the optimization reward (pLDDT>70 and RMSD<1). This is doing some kind of reward hacking, I wonder a simple protein language model + MCMC can also performs well in the task.
A1: We thank the reviewer for this insightful comments and constructive suggestions.
To addreee your concern, we have conducted experimentations of using OmegaFold during the search process, and uses ESMFold to evaluate the final designs. We also implemented PLM + MCMC following your comments. We used DPLM-650M as the base model in MCMC algorithm, as DPLM-650M is used by our ELMS framework to modify the design sequences.
| Model / System | Successful (%) | |
|---|---|---|
| Single | Paired | |
| DPLM-650M | 15.9 ± 0.4 | 6.5 ± 0.5 |
| Genie2 | 60.1 ± 1.0 | 32.9 ± 0.4 |
| La-Proteina | 67.1 ± 0.7 | 62.7 ± 0.5 |
| RFdiffusion | 65.1 ± 0.4 | 43.7 ± 1.0 |
| ELMS ( ESMfold-3B guided ) | 86.4 ± 0.8 | 84.6 ± 0.6 |
| ELMS ( OmegaFold guided ) | 71.6 ± 0.5 | 68.8 ± 0.4 |
| DPLM-650M + MCMC | 40.6 ± 0.3 | 38.6 ± 0.5 |
| DPLM-650M + MCTS | 51.9 ± 0.4 | 47.8 ± 0.7 |
-
Previous studies have shown that OmegaFold generally exhibits lower structure prediction accuracy than ESMFold (ESM-2-based)[1]. Consistent with this observation, using OmegaFold as a weaker guiding oracle reduces ELMS's success rates. The performance gap indicates some sensitivity to the choice of folding oracle, potentially reflecting differences in predictive accuracy and reward calibration. Nevertheless, the strong performance under OmegaFold-guided search and ESMFold-based evaluation provides evidence that ELMS is not exclusively dependent on ESMFold feedback. Importantly, even with this weaker and independent guiding oracle, ELMS continues to outperform all competing baselines, including La-Proteina (67.1% Single, 62.7% Paired). These results demonstrate that ELMS remains effective when different folding models are used for search and evaluation, supporting the generalizability of our framework beyond ESMFold-specific guidance.
-
We observe that DPLM-650M + MCMC or MCTS substantially improves performance over DPLM itself, confirming reviewer's instuition that evaluator-guided search is a viable baseline. However, even when granted direct access to ESMFold during search, neither baseline surpasses ELMS, further confirming ELMS's superiority.
[1]Hýsková A, Maršálková E and Šimeček P (2026) Balancing speed and precision in protein folding: a comparison of AlphaFold2, ESMFold, and OmegaFold. Front. Genet. 16:1715037. doi: 10.3389/fgene.2025.1715037
reply3
Q2: Could you help to explain the neccessity of LLM agents and how it will distinguish from deterministic rule?
A2: We sincerely thank the reviewer for raising this important and insightful question.
We agree that parsing evaluator outputs, checking success thresholds, and assigning the metric regimes in Table 10 are deterministic operations and do not require an LLM. We should distinguish the intended roles of the agents from evidence that they are necessary.
-
The Critic agent is intended to provide an evidence-grounded interpretation of the current candidate, including its failure mode and the constraints that subsequent modifications should preserve. However, because several diagnostic labels follow directly from predefined checks, producing these labels alone does not establish an advantage over a deterministic implementation.
-
The Policy agent has additional decisions beyond accepting the router’s highest-ranked operator. The router supplies a ranked, restricted action menu; the Policy selects operator/arm combinations and execution parameters, including edit scale, legal repair positions, repair fraction, temperature, and restart decisions. Its intended role is to use candidate-local evidence and search history to refine the default recommendation. The deterministic controller instead uses predefined rankings, parameter defaults, and site-selection heuristics.
To further address your concern, we compared the performance of ELMS without Critic / Policy agent. As shown in below Table, these results suggest that the Policy Agent's context-dependent action selection provides substantial benefits over the evaluated deterministic controller, while the Critic Agent contributes additional value by providing structured, state-specific diagnostic feedback.
| Model / System | Successful (%) | |
|---|---|---|
| Single | Paired | |
| ELMS | 86.4 ± 0.8 | 84.6 ± 0.6 |
| ELMS without Critic agent (Policy agent + MCTS) | 80.2 ± 0.3 | 77.4 ± 0.5 |
| ELMS without Critic and without Policy agent (deterministic rule + MCTS) | 57.9 ± 0.7 | 42.4 ± 0.4 |
| DPLM-650M + MCTS | 51.9 ± 0.4 | 47.8 ± 0.7 |
reply1
Weakness: From Table 9 we can also find that the Unique score of ELMS is extremely low compared with previous design methods like La-Proteina and RFDiffusion.
A0: We thank the reviewer for this insightful comment.
We acknowledge that the lower Unique score represents a limitation of the current framework, particularly for applications requiring a structurally diverse set of candidate scaffolds. Our current objective prioritizes discovering feasible designs under a fixed evaluation budget rather than maximizing diversity. Incorporating diversity-aware exploration and redundancy penalties into MCTS is a promising direction for improving the balance between Success, Novelty, and Unique.
Unique measures structural diversity among generated candidates, whereas Novelty measures structural dissimilarity to known reference structures. As shown in Table 9, ELMS achieves strong Success and Novelty despite its lower Unique score. This indicates that ELMS can discover scaffolds that satisfy the target constraints and differ substantially from known structures, even when its generated candidates are concentrated in fewer structural clusters.
Unlike generative models designed to sample diverse structures, ELMS formulates motif scaffolding as a bounded-budget test-time optimization problem. Through MCTS and LLM-guided structural feedback, the search preferentially allocates evaluation resources to promising trajectories. Repeated refinement along these trajectories can produce structurally similar descendants, potentially explaining the observed combination of high Success and Novelty with low Unique.