- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RosettaSearch, an inference-time protein sequence optimization framework that uses an LLM to iteratively revise sequences based on structural feedback from RosettaFold3. The search combines global fidelity metrics, residue-level error information, and multi-objective priority search, and does not require retraining the underlying protein design model. The method is evaluated on LigandMPNN and ProteinMPNN designs across PDB monomers and Dayhoff backbones, with additional checks using Chai-1 and Boltz-2, random-mutation baselines, a fine-tuned open-weight LLM, and a small wet-lab stability experiment.
Critical strengths
-
Using a general-purpose LLM as an inference-time optimizer for protein sequence design is an interesting direction. Rather than replacing ProteinMPNN or LigandMPNN, RosettaSearch treats their outputs as starting points and lets the LLM iteratively revise sequences using structural feedback. This makes the method flexible to new objectives without retraining a task-specific protein model.
-
The paper does more than show gains under the same search oracle. Improvements are also checked with Chai-1 and Boltz-2, and the method is tested on both PDB monomers and de novo Dayhoff backbones. These additional evaluations make the results more convincing than if they relied only on RosettaFold3 feedback.
Critical weaknesses
-
The evaluation does not show whether the LLM-based optimizer is competitive with specialized iterative protein-design or sequence-refinement methods. RosettaSearch is mainly compared with the initial ProteinMPNN/LigandMPNN designs and random or BLOSUM-based mutation baselines. Since the main claim is that an LLM can serve as a generative optimizer, comparisons with dedicated iterative refinement methods would be important to establish whether the LLM route offers a real advantage beyond simply allowing repeated oracle-guided updates. This is especially relevant given that RosettaSearch uses many expensive structure-prediction calls per protein.
-
It is unclear how much of the improvement comes from protein-specific reasoning by the LLM rather than sequence priors, memorization, and black-box search with a strong structure oracle. In the PDB experiments, the native sequence is explicitly provided in the LLM context, and the authors note that the model can simply return the native sequence during optimization. The Dayhoff experiment without a reference sequence is a useful control, but it does not rule out pretraining overlap or homolog memorization. A stronger contamination or homology analysis, together with results on the PDB setting without native-sequence context, would make the claimed role of LLM reasoning much more convincing.
Questions for authors
-
In the PDB monomer experiments, the native sequence is included in the LLM context, and the paper notes that the model may return the native sequence during optimization. How much of the reported gain remains if the native sequence is removed from the prompt? This would materially affect how I interpret the role of the LLM as an optimizer.
-
Was any analysis performed to estimate possible pretraining overlap or sequence homology between the evaluated proteins and the LLM’s training data? The Dayhoff setting removes the explicit reference sequence, but it does not by itself rule out memorization of related proteins or sequence families.
-
Why are specialized iterative protein-design or sequence-refinement methods not included as baselines? Since RosettaSearch uses repeated structure-oracle calls to revise sequences, comparison against dedicated iterative refiners under a matched oracle or compute budget would be important for assessing whether the LLM-based route provides a meaningful advantage.
There is no native sequence in context
We sincerely apologize for editing issues that caused some of the text from an older version of the manuscript to remain. All our current results stated in the paper DO NOT provide the native sequence in context to the LLM. The earlier versions of this work provided additional context information to aid generation, which we found to cause some reward hacking.