acceptodds
Under review as a conference paper at ICLR 2027

Evaluator-in-the-Loop Monte Carlo Tree Search Via LLM Agents for Motif Scaffolding in Protein Design

Abstract

Motif-scaffolding systems commonly follow a generate-then-filter paradigm, in which candidate proteins are generated independently and structural evaluation is used primarily for terminal screening or ranking. This paradigm underuses evaluation: failed predictions contain state-specific evidence about whether a design requires repair of motif geometry, global foldability, or other structural constraints. We introduce (Evidence-based LLM-guided Monte Carlo Search), an evaluator-in-the-loop search framework for motif scaffolding that turns such evaluator feedback into targeted design actions. Effective reuse of structural feedback is nontrivial because different scaffold states exhibit different failure modes, and repeatedly refining a single trajectory can prematurely commit computation to an unproductive region of sequence space. ELMS therefore retains evaluated scaffolds as persistent search states: a Critic Agent diagnoses state-local structural failures, a Policy Agent selects targeted operators with execution parameters, motif-locked operators realize legal sequence modifications, and MCTS determines which historical states should receive further design effort. Under the standard GeomMotif protocol (100 candidates per task), ELMS achieves Successful rates of 86.41% on single-motif tasks and 84.57% on paired-motif tasks, exceeding the strongest prior baseline by 19.3 and 21.9 percentage points, respectively. On MotifBench, under a matched 100-candidate search budget, it solves 26.7 of 30 tasks on average (88.89% Task Success), compared with 16.0 tasks (53.33%) for the strongest baseline. These results establish ELMS as an effective approach for converting structural evaluation from a terminal filter into actionable guidance for iterative motif scaffolding.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein design, motif scaffolding, geometric preservation, deep learning, Large Language Models, structural biology, Multi-Agent System, Monte Carlo Tree Search

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U5yeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced a framework named ELMS which utilizes Monte Carlo Tree Search along with LLM and evaluator feedback for motif scaffolding. The method is evaluated on GeomMotif and MotifBench under a fixed budget and reports substantially higher task success rates than existing generative baselines.

Critical strengths
  1. The paper proposed a useful solution for motif scaffolding, using the evaluator feedback as signal rather than using generate-filter process.
  2. The system is carefully modularized into non-LLM and LLM part. The authors provide implementation details, which improve the clarity and reproducibility of the proposed framework.
Critical weaknesses
  1. ESMFold is used during optimization and evaluation, expecially on the GeomMotif task. The final criterion is nearly the same as the optimization reward (pLDDT>70 and RMSD<1). This is doing some kind of reward hacking, I wonder a simple protein language model + MCMC can also performs well in the task. From Table 9 we can also find that the Unique score of ELMS is extremely low compared with previous design methods like La-Proteina and RFDiffusion. I suggest that the authors should evaluate the final designs with an independent structure predictor, though these structure prediction models might show similar distribution.
  2. As I said in 1., I wonder how simple PLM+MCMC might perform under the same setting. I noticed there is greedy search baseline implemented but I still wonder some other standard evaluator-guided search methods under the same evaluation budget.
  3. The contribution of MCTS is difficult to isolate. The key idea of the paper should be MCTS, while the method introduces many additional components, including adaptive operators and static action arms. How are these components chosen, and how much of the improvement comes from these hand-designed priors rather than MCTS itself?
  4. The necessity of the LLM agents is unclear. It looks like the Critic Agent is mainly doing parsing work (the problem is actually deterministic according to Table 10) and the Policy Agent selects from operators already ranked by the operator router.
Questions for authors
  1. I would like to see some simple evaluator-guided method and using seperate protein folding tool in optimization and evaluation.
  2. Could you help to explain the neccessity of LLM agents and how it will distinguish from deterministic rule?
5 comments

Weakness: From Table 9 we can also find that the Unique score of ELMS is extremely low compared with previous design methods like La-Proteina and RFDiffusion.

A0: We thank the reviewer for this insightful comment.

We acknowledge that the lower Unique score represents a limitation of the current framework, particularly for applications requiring a structurally diverse set of candidate scaffolds. Our current objective prioritizes discovering feasible designs under a fixed evaluation budget rather than maximizing diversity. Incorporating diversity-aware exploration and redundancy penalties into MCTS is a promising direction for improving the balance between Success, Novelty, and Unique.

  • Unique measures structural diversity among generated candidates, whereas Novelty measures structural dissimilarity to known reference structures. As shown in Table 9, ELMS achieves strong Success and Novelty despite its lower Unique score. This indicates that ELMS can discover scaffolds that satisfy the target constraints and differ substantially from known structures, even when its generated candidates are concentrated in fewer structural clusters.

  • Unlike generative models designed to sample diverse structures, ELMS formulates motif scaffolding as a bounded-budget test-time optimization problem. Through MCTS and LLM-guided structural feedback, the search preferentially allocates evaluation resources to promising trajectories. Repeated refinement along these trajectories can produce structurally similar descendants, potentially explaining the observed combination of high Success and Novelty with low Unique.

Q1: I would like to see some simple evaluator-guided method and using seperate protein folding tool in optimization and evaluation. + Weakness: I suggest that the authors should evaluate the final designs with an independent structure predictor, though these structure prediction models might show similar distribution. + Weakness: ESMFold is used during optimization and evaluation, expecially on the GeomMotif task. The final criterion is nearly the same as the optimization reward (pLDDT>70 and RMSD<1). This is doing some kind of reward hacking, I wonder a simple protein language model + MCMC can also performs well in the task.

A1: We thank the reviewer for this insightful comments and constructive suggestions.

To addreee your concern, we have conducted experimentations of using OmegaFold during the search process, and uses ESMFold to evaluate the final designs. We also implemented PLM + MCMC following your comments. We used DPLM-650M as the base model in MCMC algorithm, as DPLM-650M is used by our ELMS framework to modify the design sequences.

Model / SystemSuccessful (%)
SinglePaired
DPLM-650M15.9 ± 0.46.5 ± 0.5
Genie260.1 ± 1.032.9 ± 0.4
La-Proteina67.1 ± 0.762.7 ± 0.5
RFdiffusion65.1 ± 0.443.7 ± 1.0
ELMS ( ESMfold-3B guided )86.4 ± 0.884.6 ± 0.6
ELMS ( OmegaFold guided )71.6 ± 0.568.8 ± 0.4
DPLM-650M + MCMC40.6 ± 0.338.6 ± 0.5
DPLM-650M + MCTS51.9 ± 0.447.8 ± 0.7
  • Previous studies have shown that OmegaFold generally exhibits lower structure prediction accuracy than ESMFold (ESM-2-based)[1]. Consistent with this observation, using OmegaFold as a weaker guiding oracle reduces ELMS's success rates. The performance gap indicates some sensitivity to the choice of folding oracle, potentially reflecting differences in predictive accuracy and reward calibration. Nevertheless, the strong performance under OmegaFold-guided search and ESMFold-based evaluation provides evidence that ELMS is not exclusively dependent on ESMFold feedback. Importantly, even with this weaker and independent guiding oracle, ELMS continues to outperform all competing baselines, including La-Proteina (67.1% Single, 62.7% Paired). These results demonstrate that ELMS remains effective when different folding models are used for search and evaluation, supporting the generalizability of our framework beyond ESMFold-specific guidance.

  • We observe that DPLM-650M + MCMC or MCTS substantially improves performance over DPLM itself, confirming reviewer's instuition that evaluator-guided search is a viable baseline. However, even when granted direct access to ESMFold during search, neither baseline surpasses ELMS, further confirming ELMS's superiority.

[1]Hýsková A, Maršálková E and Šimeček P (2026) Balancing speed and precision in protein folding: a comparison of AlphaFold2, ESMFold, and OmegaFold. Front. Genet. 16:1715037. doi: 10.3389/fgene.2025.1715037

Q2: Could you help to explain the neccessity of LLM agents and how it will distinguish from deterministic rule?

A2: We sincerely thank the reviewer for raising this important and insightful question.

We agree that parsing evaluator outputs, checking success thresholds, and assigning the metric regimes in Table 10 are deterministic operations and do not require an LLM. We should distinguish the intended roles of the agents from evidence that they are necessary.

  • The Critic agent is intended to provide an evidence-grounded interpretation of the current candidate, including its failure mode and the constraints that subsequent modifications should preserve. However, because several diagnostic labels follow directly from predefined checks, producing these labels alone does not establish an advantage over a deterministic implementation.

  • The Policy agent has additional decisions beyond accepting the router’s highest-ranked operator. The router supplies a ranked, restricted action menu; the Policy selects operator/arm combinations and execution parameters, including edit scale, legal repair positions, repair fraction, temperature, and restart decisions. Its intended role is to use candidate-local evidence and search history to refine the default recommendation. The deterministic controller instead uses predefined rankings, parameter defaults, and site-selection heuristics.

To further address your concern, we compared the performance of ELMS without Critic / Policy agent. As shown in below Table, these results suggest that the Policy Agent's context-dependent action selection provides substantial benefits over the evaluated deterministic controller, while the Critic Agent contributes additional value by providing structured, state-specific diagnostic feedback.

Model / SystemSuccessful (%)
SinglePaired
ELMS86.4 ± 0.884.6 ± 0.6
ELMS without Critic agent (Policy agent + MCTS)80.2 ± 0.377.4 ± 0.5
ELMS without Critic and without Policy agent (deterministic rule + MCTS)57.9 ± 0.742.4 ± 0.4
DPLM-650M + MCTS51.9 ± 0.447.8 ± 0.7
Reviewer_WdUEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposes evaluator-in-the-loop Monte Carlo tree search to enhance motif scaffolding in protein design. It leverages a Critic Agent and a Policy Agent to instruct a protein language model to iteratively editing process to fulfill the process. It achieves substantial improvement on GeomMotif and MotifBench against competitive models.

Critical strengths
  • The paper addresses an important weakness of generate-then-filter protein design, which is well-motivated.
  • The method is novel and well-designed.
  • The results are strong and promising on both of GeomMotif and MotifBench.
Critical weaknesses
  • ESMFold is used both search and evaluation which leads to the risk that the algorithm can overfit ESMFold.
  • Lack of detailed analysis on LLM contribution.
Questions for authors
  • Please provide a representative case to show the search trajectory with Critic Agent and Policy Agent.
  • Can you show the results when different folding models are adopted in search and evaluation process? Is the algorithm generalizable to different folding model?
  • Please report the runtime budget of the proposed algorithm and competitive models.
5 comments

Q1: Please provide a representative case to show the search trajectory with Critic Agent and Policy Agent.

A1: Thank you for raising this point.

We provide a detailed search trajectory of our framework on Task7: 5OJ8 in Geommotif Benchmark here.

The following trajectory represents a selected parent-child path extracted from the MCTS search tree, rather than the complete set of evaluated candidates.

Task7 / 5OJ8

Step 1
Current: pLDDT 36.0282; RMSD 2.1676 Å.
Critic output: Both gates fail: incorrect relative motif placement, connector routing, and weak downstream scaffold. Positions: 37–51; 62–78.
Policy output: Rule 4: scale matched refinement; arms junction closure (37–51), helix bundle (62–78).
Implemented modification: 37–51: PNLMFVDWFEHQWLE → AESLAKGALSAELKA; 62–78: GRFWDRDWLYDNKTHWF → ALASEALKAVSALEAGL.
Result: pLDDT 41.8557; RMSD 1.0134 Å.

Step 2
Current: pLDDT 41.8557; RMSD 1.0134 Å.
Critic output: Confidence is low and joint motif RMSD narrowly fails. Test connector shortening and distal scaffold repair. Positions: 33–40 (delete); 195–218.
Policy output: Rule 1: masked motif constraint router; arms junction closure (33–40), helix bundle (195–218).
Implemented modification: Delete 33–40; 195–218: FWFTQKDPSVTRYHYKSWFKESMR → ALAELAELAELAELAELGAELAEL.
Result: pLDDT 46.0411; RMSD 0.8011 Å; length 250 → 242.

Step 3
Current: pLDDT 46.0411; RMSD 0.8011 Å.
Critic output: Geometry passes; the middle/distal suffix remains low-confidence. Preserve motif geometry. Positions: 93–123.
Policy output: Rule 3: esmfold oracle calibration; arm helix bundle (93–123).
Implemented modification: 93–123: VFPAQQPEGASPKVDAFPLNGNPIVFLYPWC → ILSELKVGTLAELASRIEVGSTALEKILGSA.
Result: pLDDT 56.4068; RMSD 0.6085 Å.

Step 4
Current: pLDDT 56.4068; RMSD 0.6085 Å.
Critic output: Geometry passes; a weak distal transition limits global confidence. Positions: 124–154.
Policy output: Rule 4: scale matched refinement; arm helix bundle (124–154).
Implemented modification: 124–154: CNWFKDVRQFTVICGIKAPTQRWTPTKWMTY → IAEKVLASKLEAKVLSGLEAKVASKLEAKVL.
Result: pLDDT 54.6021; RMSD 0.6529 Å.

Step 5
Current: pLDDT 54.6021; RMSD 0.6529 Å.
Critic output: Geometry passes; the distal suffix remains poorly organized and low-confidence. Positions: 155–185.
Policy output: Rule 15: robust confidence screen; arm helix bundle (155–185).
Implemented modification: 155–185: AARWTKMPEWMPHYLRLYGRCWWMPEHEDQP → LEALKALSGTAKLAEVASTALEKILGSATAL.
Result: pLDDT 58.0752; RMSD 0.6522 Å.

Step 6
Current: pLDDT 58.0752; RMSD 0.6522 Å.
Critic output: Geometry passes; the unedited distal transition remains a confidence bottleneck. Positions: 186–216.
Policy output: Rule 3: esmfold oracle calibration; arm helix bundle (186–216).
Implemented modification: 186–216: TALAELAELAELAELAELGAELAELCEFSAR → LEALKALSGTAKLAEVASTLAEKIKLGSATA.
Result: pLDDT 62.7236; RMSD 0.7268 Å.

Step 7
Current: pLDDT 62.7236; RMSD 0.7268 Å.
Critic output: Geometry passes; weak distal structure and uncertain connector routing motivate a layout test. Positions: before 44 (insert 8 aa); 220–242 (delete).
Policy output: Rule 9: confidence geometry plateau test; arms junction closure (before 44), minimal frustration (220–242).
Implemented modification: Insert PGASGTAL before 44; delete 220–242.
Result: pLDDT 69.2626; RMSD 0.9713 Å; length 242 → 227.

Step 8
Current: pLDDT 69.2626; RMSD 0.9713 Å.
Critic output: Confidence narrowly fails; the connector junction is weak and the geometry margin is small. Preserve length. Positions: 37–42.
Policy output: Rule 8: near threshold trust region; arm junction closure (37–42).
Implemented modification: 37–42: LSAELK → GPSAAE.
Result: pLDDT 70.3858; RMSD 0.9616 Å.

Final: Success+Novel. pLDDT 70.3858; RMSD 0.9616 Å; offline native-normalized TM-score 0.56602.

Q2: Can you show the results when different folding models are adopted in search and evaluation process? Is the algorithm generalizable to different folding model? Weakness: ESMFold is used both search and evaluation which leads to the risk that the algorithm can overfit ESMFold.

A2: Thank you for raising this important point.

To address concerns regarding whether the framework exploits biases specific to ESMFold since it is used in both search and evaluation, we evaluated our framework using OmegaFold[1] as the guiding oracle during the search phase, while maintaining ESM-3b as the final external evaluation benchmark.

Model / SystemSuccessful (%)
SinglePaired
Genie260.1 ± 1.032.9 ± 0.4
La-Proteina67.1 ± 0.762.7 ± 0.5
RFdiffusion65.1 ± 0.443.7 ± 1.0
ELMS ( ESMfold-3B guided )86.4 ± 0.884.6 ± 0.6
ELMS ( OmegaFold guided )71.6 ± 0.568.8 ± 0.4

Previous studies have shown that OmegaFold generally exhibits lower structure prediction accuracy than ESMFold (ESM-2-based)[1]. Consistent with this observation, using OmegaFold as a weaker guiding oracle reduces ELMS's success rates. The performance gap indicates some sensitivity to the choice of folding oracle, potentially reflecting differences in predictive accuracy and reward calibration. Nevertheless, the strong performance under OmegaFold-guided search and ESMFold-based evaluation provides evidence that ELMS is not exclusively dependent on ESMFold feedback. Importantly, even with this weaker and independent guiding oracle, ELMS continues to outperform all competing baselines, including La-Proteina (67.1% Single, 62.7% Paired). These results demonstrate that ELMS remains effective when different folding models are used for search and evaluation, supporting the generalizability of our framework beyond ESMFold-specific guidance.

Q3: Please report the runtime budget of the proposed algorithm and competitive models.

A3: We evaluated the computational efficiency of our framework against competing models across all 57 GeomMotif benchmark tasks under identical hardware conditions using two NVIDIA A6000 GPUs. All methods were evaluated with the same candidate budget and parallelization settings. The reported runtime represents the end-to-end wall-clock time, including LLM API latency, PLM-based sequence generation, structure prediction (folding), and all other computational overhead.:

MethodRuntime for 57 Tasks (h)
ELMS33.31
Genie291.04
La-Proteina31.31

[1]Hýsková A, Maršálková E and Šimeček P (2026) Balancing speed and precision in protein folding: a comparison of AlphaFold2, ESMFold, and OmegaFold. Front. Genet. 16:1715037. doi: 10.3389/fgene.2025.1715037

Weakness: Lack of detailed analysis on LLM contribution.

A: We thank the reviewer for highlighting the need to better quantify the contribution of LLM agents. To address this concern, we conducted additional ablation experiments on the 57 GeomMotif tasks, systematically removing the Critic and Policy agents while retaining MCTS.

Model / SystemSuccessful (%)
SinglePaired
ELMS86.4 ± 0.884.6 ± 0.6
ELMS without Critic agent (Policy agent + MCTS)80.2 ± 0.377.4 ± 0.5
ELMS without Critic and without Policy agent (deterministic rule + MCTS)57.9 ± 0.742.4 ± 0.4
DPLM-650M + MCTS51.9 ± 0.447.8 ± 0.7
  • Contribution of Policy Agent: Replacing deterministic rule-based action selection with the LLM Policy Agent substantially improves performance on both Single and Paired tasks, particularly on the more challenging Paired tasks. This suggests that adaptive operator selection informed by structural feedback and search history is considerably more effective than following predefined routing heuristics.

  • Contribution of Critic Agent: Removing the Critic Agent while retaining the Policy Agent and MCTS consistently reduces performance across both task categories. This indicates that the Critic Agent provides additional value by interpreting candidate-specific structural failures and guiding subsequent modifications, complementing the Policy Agent's decision-making.

Overall, these results demonstrate the value of both LLM agents, with the Policy Agent providing the larger improvement and the Critic Agent offering additional benefits within the MCTS framework.

We hope our additional experiments and analysis address your concerns.

Reviewer_AAyjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose ELMS, an LLM-guided Monte Carlo Tree Search (MCTS) framework for protein motif scaffolding. Instead of independently generating and filtering candidate designs, ELMS uses ESMFold feedback to iteratively improve protein sequences. A Critic Agent diagnoses structural failures, a Policy Agent selects sequence-editing operators, and MCTS maintains multiple search trajectories to allocate subsequent evaluations. The framework is evaluated on GeomMotif and MotifBench, where the authors report substantial improvements over existing generative methods and simpler search strategies.

Critical strengths
  1. Motif scaffolding is an important and challenging problem. Reusing structural evaluation feedback to guide subsequent sequence generation, rather than relying entirely on independent sampling, is a potentially useful direction.
  2. The paper evaluates the proposed method on both GeomMotif and MotifBench and includes comparisons with several established protein design methods.
Critical weaknesses
  1. Some critical experiment details are lacking. For example, what protein language model (PLM) was used for sequence generation? More importantly, the actual sequence-editing operations are not sufficiently defined. Although Appendix C.4 lists 19 adaptive operator identifiers and 12 static action arms, it primarily describes their intended purposes rather than their executable implementations. For example, the manuscript does not explain how residues are selected and regenerated during partial renoising, how parent sequences and crossover positions are selected during recombination, or which model is used for backbone-conditioned inverse folding.
  2. There is a substantial discrepancy in the MotifBench baseline results. Table 3 reports that RFdiffusion3 solves only 15.7/30 tasks on average (52.22%), whereas the publicly released RFdiffusion3 MotifBench evaluations (https://github.com/lorenzkleiter/RFD3-Motifbench-Fixed) report 28, 27, and 27 solved tasks across three independent runs, each using 100 backbones per task. The supplementary material states that baseline results are converted from public evaluation packages to a 100-candidate endpoint, but the candidate-selection procedure and reasons for this discrepancy are not sufficiently explained.
Questions for authors
  1. Which protein models are actually used for sequence generation? Does ELMS use a single protein sequence model or multiple models? For the candidate backbone inverse-folding operator, which specific inverse-folding model is used?
  2. How are the 19 adaptive operators and 12 static action arms implemented? Please provide executable definitions or pseudocode for the sequence-editing operations. How are search-management operators, such as Pareto archiving, represented as MCTS actions and assigned rewards?
  3. How many candidates are generated, evaluated, and finally submitted under the 100-candidate search budget? Since ELMS will generate a search trajectory, so it will be great to report the number of raw generated sequences, rejected or duplicate candidates, actual ESMFold calls, and final retained designs per task.
  4. How are the reported MotifBench baseline results obtained?
  5. How much improvement comes from the underlying PLM, the action library, LLM routing, and MCTS, respectively? A particularly important baseline would be to generate 100 independent sequences directly from the exact same PLM used by ELMS. The authors should also compare ELMS against a rule-based or non-LLM operator router using the same action library and MCTS.
  6. How sensitive is ELMS to the choice of LLM? The current implementation uses GPT-5-nano, will the success rate further improve if we choose a better LLM, such as GPT-6.1-Sol?
  7. Can the authors report the official MotifBench Score for the MotifBench results? This will enable an easy comparison with other leaderboard methods.
4 comments

We thank you for bringing this up

Q1.1: Which protein models are actually used for sequence generation?

A1.1: We use DPLM-650M because, while standalone DPLM-650M faces limitations in direct motif scaffolding (as shown in Table 2), its probabilistic generative capacity makes it an ideal engine for iterative sequence modification and refinement when guided by our MCTS and static arms.

Q1.2: Does ELMS use a single protein sequence model or multiple models?

A1.2 ELMS relies primarily on DPLM-650M as its core protein sequence generative model, while incorporating ProteinMPNN specifically as an auxiliary inverse-folding operator for backbone-conditioned candidate refinement.

Q1.3: For the candidate backbone inverse-folding operator, which specific inverse-folding model is used?

A1.3: The candidate-backbone inverse-folding operator uses ProteinMPNN with the v_48_020 checkpoint. It conditions on the ESMFold-predicted backbone of a previously evaluated candidate and proposes scaffold-sequence edits while keeping the constrained motif residues fixed. The resulting child candidates are subsequently evaluated with ESMFold.

Q2.1: How are the 19 adaptive operators and 12 static action arms implemented? Please provide executable definitions or pseudocode for the sequence-editing operations.

A2.1 We thank the reviewer for bringing this up. Appendix C.4 defines the 19 identifiers as immutable search priors, and Appendix E describes their realization through motif-locked sequence operations. We agree that the mapping from these identifiers to executable kernels, together with the numerical definitions of the static arms, needs to be made more explicit. We will expand this mapping and distinguish implemented controller rules, sequence-editing kernels, and optional extensions.

The protein sequence model used in the reported search was DPLM-650M, as clarified here.

Following Sections 4.1–4.3, MCTS selects an evaluated parent, the Critic interprets its cached evidence, and the Policy selects legal action parameters. Below we expand the DPLM action-realization stage described in Appendix E. Let FF be the fixed motif positions and UU the mutable scaffold positions. An action specifies an operator identifier, arm, search scale, repair positions, repair fraction, temperature, and requested child count.

  1. Full restart. All positions in UU are masked, while the prescribed motif residues remain fixed. DPLM generates a completion conditioned on the public motif mask.

  2. Partial renoising. For a parent sequence xx, the standard implementation uses k=max⁡(1,⌈ρ∣U∣⌉)k=\max(1,\lceil\rho|U|\rceil) editable positions, with ρ\rho clipped to [0.10,0.35][0.10,0.35]. Legal policy-requested positions are followed by candidate-local low-confidence positions, motif-adjacent scaffold positions, and remaining mutable positions. This ordered list is deduplicated and capped at 60 sites. For junction_closure, internal-gap boundary priorities are prepended. The first kk eligible sites are selected; any remaining slots are filled by seeded uniform sampling without replacement. Selected sites are masked and decoded, while unselected parent residues remain fixed. Resampling can recover an original residue, so masked-site count and realized edit count must be distinguished; the candidate metadata records realized edit positions.

  3. Single-site refinement. Candidate repair positions are obtained from the same legal priority rules. Each position is masked separately, and DPLM supplies conditional log-probabilities over the 20 canonical amino acids. The existing residue is excluded. For a requested batch of BB children, the ordinary path considers up to the first max⁡(4,B)\max(4,B) prioritized sites and retains up to min⁡(8,max⁡(2,B))\min(8,\max(2,B)) non-parent alternatives per site. Proposed substitutions are ranked first by increasing log-probability of the existing parent residue and then by decreasing log-probability of the replacement. Each retained child changes exactly one residue. This local path does not apply static-arm logit biases. The supplied source also specifies the larger-batch diagnostic ordering and optional exact-site mode.

For restart and partial-renoising actions, each static arm supplies the deterministic additive bias defined in Appendix E:

a^i=argmax⁡a∈A[ℓθ(a∣u)i+bh(i,a;u,μ)+τgi,a]\hat{a}_i = \underset{a \in \mathcal{A}}{\operatorname{argmax}} \left[ \ell_{\theta}(a \mid u)_i + b_h(i,a;u,\mu) + \tau g_{i,a} \right]

where uu is the current token sequence, μ\mu is the public motif mask, and gi,ag_{i,a} is independent standard Gumbel noise. The decoder permits only canonical amino-acid tokens and uses the stated default of 100 decoding iterations. These are sequence-decoding steps, not additional ESMFold evaluations. The remasked set decreases over these iterations, with unselected parent context and fixed motif residues held constant.

A2.1 continued

The 12 arms are plm_only, helix_bundle, beta_core, beta_edge_turn, globular_balance, minimal_frustration, junction_closure, transmembrane_bundle, charge_balanced_globular, segmented_helix_bundle, alpha_repeat, and alpha_beta_layer, matching Table 11. plm_only contributes zero arm bias; the other arms add explicit residue-group, positional-pattern, boundary, and context-dependent biases. For example, helix_bundle adds 0.42 to A/L/E/K/Q/M, subtracts 1.35 from P and 0.55 from G, and adds a heptad-dependent hydrophobic/polar preference. Numerical coefficients are fixed during search.

The following pseudocode preserves the manuscript's Critic–Policy flow and expands one DPLM action.

parent = MCTS.select()
critique = Critic(parent.cached_evidence)
decision = validate(
    Policy(parent.public_state, parent.cached_evidence, critique, legal_menu)
)

# Each action alpha contains its rule, arm, scale, sites, fraction,
for alpha in materialize_validated_actions(decision):
    scale = resolve_scale(alpha, parent.feedback, branch_diagnostics)

    if scale == "restart":
        S = U
        context = public_motif_mask
        sequences = DPLM_decode(context, S, alpha.arm, alpha.temperature)

    elif scale == "medium":
        rho = clip(alpha.repair_fraction, 0.10, 0.35)
        k = max(1, ceil(rho * len(U)))
        P = ordered_unique(
            legal_requested_sites
            + low_confidence_sites
            + motif_boundary_sites
            + remaining_mutable_sites
        )[:60]
        if alpha.arm == "junction_closure":
            P = ordered_unique(internal_gap_boundary_priority + P)
        S = first_k(P)
        fill any deficit by seeded sampling without replacement from U minus S
        sequences = DPLM_decode(parent.sequence, S, alpha.arm, alpha.temperature)

    else: 
        P = prioritized_legal_sites(parent.feedback, alpha)
        substitutions = DPLM_rank_nonparent_substitutions(parent.sequence, P)
        sequences = apply_one_substitution_per_child(substitutions)

    for sequence in valid_unseen(sequences):
        child = record_candidate(sequence, parent, alpha, actual_proposal_kind)
        feedback = ESMFold_evaluate(child)
        R = feasibility_proxy_reward(feedback)  # Appendix D
        edge = MCTS.attach(parent, child, feedback)
        MCTS.backpropagate(edge.path, R)

Q2.2: How are search-management operators, such as Pareto archiving, represented as MCTS actions and assigned rewards?

A2.2: Thank you for highlighting this ambiguity. Search-management rules are not represented as standalone MCTS transitions. Each non-root node corresponds to a protein candidate, and each edge records a realized sequence proposal together with its rule_id, arm, and proposal_kind. A management-rule identifier can annotate such an edge, but selecting the identifier does not create an additional archive-only node.

Rewards are assigned from the evaluated candidate’s structural feedback. Specifically, the child receives the feasibility-proxy reward defined in Appendix D, based on pLDDT and motif RMSD. This reward is backed up along the selected tree path:

W_e \leftarrow W_e + R_j,\qquad Q_e = W_e/N_e.$$ The same return updates the empirical statistics of each distinct rule identifier encountered along that path, once per identifier per evaluated child. Consequently, a management-rule label receives credit through the measured outcomes of sequence proposals carrying that label. There is no separate reward for retaining a candidate, updating search history, increasing archive size, or improving Pareto hypervolume. History maintenance itself does not trigger an additional visit-count increment or reward update. We also acknowledge that the name `pareto_basin_archive` overstates its implementation if interpreted as a dedicated Pareto-optimization action. In the inspected implementation, it is a controller-level routing heuristic intended to encourage consideration of candidates with different confidence–geometry trade-offs; the tree retains evaluated candidate history, but there is no separate nondominated-front update kernel associated with this identifier. We will revise Appendix C.4 to distinguish these management heuristics from executable sequence-editing operators and make their credit-assignment semantics explicit.

Q3: How many candidates are generated, evaluated, and finally submitted under the 100-candidate search budget? Since ELMS will generate a search trajectory, so it will be great to report the number of raw generated sequences, rejected or duplicate candidates, actual ESMFold calls, and final retained designs per task.

A3: We thank the reviewer's suggestion, and we have reported the number here.

ItemTotal across 30 tasks of Motifbench
raw sequence outputs4,434
rejections1,434
ESMFold evaluations3000
Final retained designs3000
Successful candidates685
Success+Novel candidates561

Q4: How are the reported MotifBench baseline results obtained?

A4: We used the unindexed RFdiffusion3 results from the publicly released RFD3-Unindexed-MotifBench repository. The indexed variant achieves higher performance but uses manually specified motif indices. Our reported baseline therefore corresponds to a different configuration from the indexed results cited by the reviewer.

To standardize the comparison at a 100-candidate budget per task, we defined one budget unit as a generated backbone paired with one ProteinMPNN-designed sequence. The public evaluation produces 100 backbones per task, with eight ProteinMPNN sequences per backbone, yielding 800 backbone–sequence candidates. We uniformly sampled 100 candidates from this pool and computed the reported baseline metrics on the sampled subset.

Thus, our results represent a post hoc, candidate-level budget normalization of the public evaluations. The use of the unindexed configuration and the reduction from 800 to 100 candidates account for differences from results reported using the full indexed evaluation.

Q5: How much improvement comes from the underlying PLM, the action library, LLM routing, and MCTS, respectively? A particularly important baseline would be to generate 100 independent sequences directly from the exact same PLM used by ELMS. The authors should also compare ELMS against a rule-based or non-LLM operator router using the same action library and MCTS.

A5: Thank you for suggesting these controls. We conducted supplementary experiments comparing independent sampling from DPLM-650M with a non-LLM rule-based controller and an LLM Policy controller. Both controller-based conditions use the same action library, DPLM executor, MCTS implementation, and structural reward.

All conditions were evaluated on the 57 GeomMotif tasks using three matched seeds and a budget of 100 charged evaluator-query slots per task-seed, including initialization. This corresponds to 17,100 evaluator-query slots per condition across 171 task-seed pairs.

ConditionSingle Success (%)Paired Success(%)
DPLM-650M15.9 ± 0.46.5 ± 0.5
Rule-based routing + MCTS57.9 ± 0.742.4 ± 0.4
LLM Critic + Policy + MCTS86.4 ± 0.884.6 ± 0.6

Candidate success is reported as the mean ± sample standard deviation across three seeds.

These results demonstrate two findings.

  • First, structured iterative search substantially improves over independent sampling from the same underlying PLM.
  • Second, replacing rule-based routing with LLM-guided routing, while keeping the action library, executor, MCTS, and reward fixed, further improves success rates by 28.5 percentage points on single-motif tasks and 42.2 percentage points on paired-motif tasks.

This provides direct evidence that LLM-guided action selection contributes beyond the underlying PLM and search infrastructure.

Q6: How sensitive is ELMS to the choice of LLM? The current implementation uses GPT-5-nano, will the success rate further improve if we choose a better LLM, such as GPT-6.1-Sol?

A6: Thank you for raising this question. To assess ELMS’s sensitivity to the choice of LLM, we compare its success rates on single-motif and paired-motif tasks using GPT-5-nano and GPT-6.1-Sol.

LLMSingle-Motif Success Rate (%)Paired-Motif Success Rate (%)
GPT-5-nano86.484.6
GPT-6.1-Sol85.785.1

Both LLMs achieve similar success rates, with no consistent improvement from GPT-6.1-Sol across the two task types.

Q7: Can the authors report the official MotifBench Score for the MotifBench results? This will enable an easy comparison with other leaderboard methods.

A7: Thank you for requesting the official MotifBench Score. We agree that this metric is important for comparison with leaderboard methods. We would also like to clarify how the concentration of our generated candidates affects the interpretation of our results.

Success, novelty, and structural diversity capture distinct aspects of performance. As shown in Table 9, ELMS achieves strong computational success and novelty results on GeomMotif, while its Unique values are substantially lower. A similar concentration of successful candidates into relatively few structural clusters is observed in our MotifBench outputs.

The motifbench of our framework is 22.79

A plausible explanation is the combined influence of the fixed action library, LLM-guided action selection, and MCTS allocation of the evaluation budget. ELMS uses structural feedback to identify promising candidates and preferentially expands branches that approach or satisfy the online feasibility criteria. Repeated refinement of these branches can produce multiple descendants with shared search ancestry and similar predicted structures. This provides a plausible mechanism for obtaining many successful candidates within relatively few structural clusters, although the current experiments do not establish the separate causal contribution of each component.

This concentration is a meaningful limitation. MotifBench rewards the number of structurally distinct successful solutions per task, so additional successful candidates within an existing cluster do not increase the unique-solution count. We therefore do not interpret a high candidate success rate as evidence of broad structural coverage. Likewise, a candidate can satisfy the benchmark's novelty criterion while remaining structurally similar to other candidates generated by ELMS.

Nevertheless, limited diversity does not negate the ability to discover successful, novel candidates under the stated computational evaluation criteria. Our contribution is the use of evaluator feedback to guide candidate discovery and refinement under a bounded evaluation budget. Families of related candidates may provide useful starting points for focused experimental optimization, but their practical value and experimental success remain to be established.

We will make this limitation and the scope of our contribution explicit in the revised manuscript. Improving the balance between refinement and exploration is an important direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.