acceptodds
Under review as a conference paper at ICLR 2027

PROTEAN: Beyond De Novo Generation — Can LLM Agents Perform Text-Guided Protein Structure Editing?

Abstract

Generative AI has achieved remarkable success in de novo protein design, yet protein engineering typically begins with existing, characterized proteins. The primary demand is targeted local editing, e.g., engineering a specific region to optimize affinity, alter immunogenicity, or improve stability, while strictly preserving core catalytic motifs and overall scaffold integrity. However, no standardized framework currently exists for carrying out and evaluating such edits. In current generative-design workflows, researchers often need to manually specify complex low-level parameters and rely on repetitive, trial-and-error sampling and evaluation across generative backends to search for viable conformations. While large language models (LLMs) offer an intuitive interface to express these goals using natural language, prevailing structure-generation models cannot directly interpret high-level human instructions. To bridge this gap, we introduce PROTEAN, a standardized benchmark for text-guided protein structural editing. PROTEAN comprises 380 sub-tasks spanning six intervention families and evaluates a two-stage pipeline in which an LLM translates natural-language goals into backend-specific geometric constraints and a generative model remodels the protein backbone. A two-tier evaluation separates geometric compliance, including target attainment and off-target scope fidelity, from biophysical realizability assessed by inverse folding and refolding consistency. Across five frontier LLMs paired with three backends, strict end-to-end biophysical success starts at just 7.0%. Ablation studies show that LLM constraint compilation is already highly accurate, whereas the main bottlenecks are backend-specific scope leakage and refolding failure. PROTEAN provides an open testbed for developing autonomous, multi-backend macromolecular design agents.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
text-guided protein editing, protein structure editing, LLM agents, benchmark, protein structure generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KaNVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • PROTEAN addresses the problem of targeted protein structure editing, where specific regions need to be modified while preserving functional motifs and the surrounding structural scaffold. It proposes a two-stage pipeline in which an LLM translates natural-language instructions into backend-specific JSON specifications, which are then executed by protein-design models. The authors introduce a benchmark of 380 tasks covering eight types of structural editing and evaluate five LLMs across three backends: BoltzGen, ODesign, and RFdiffusion3.

  • Experiments show that Claude-Opus-5 achieves the highest joint geometric compliance of 46.0% under pass@16 for length-preserving editing tasks. The analysis identifies structural scope leakage, spatial-planning errors, and refolding failures as major bottlenecks. Although the framework demonstrates the potential of using LLMs to orchestrate specialized protein-design models, the results also reveal substantial differences in how models translate structural objectives into backend parameters. A more detailed comparison of their decision-making processes could further clarify why some models produce more effective editing specifications than others.

Critical strengths
  • PROTEAN moves beyond de novo protein generation to address targeted editing of existing protein structures. Its benchmark comprises 380 tasks across eight editing types, with explicit constraints on editable regions and structural preservation.

  • By evaluating five LLMs across three generative backends, the study distinguishes geometric compliance from biophysical realizability, providing a more comprehensive assessment of whether generated structures satisfy editing requirements and remain physically plausible.

  • Detailed analyses of hotspot selection and diffusion-region specification reveal differences in how LLMs translate structural instructions into backend parameters. The findings highlight scope leakage and multi-constraint coordination as important challenges for future protein-editing agents.

Critical weaknesses

The paper does not sufficiently analyze the reasoning behind LLMs’ failure cases and backend parameter choices, limiting the understanding of why models differ in performance (protein-structure understanding, spatial planning, constraint coordination, or the interpretation of backend-specific parameters) and how their capabilities can be improved.

I understand that obtaining explicit reasoning traces from closed-source LLMs may be difficult. Nevertheless, the authors could consider alternatives, such as collecting structured decision rationales or recording intermediate parameter-selection decisions.

Questions for authors
  • Section 4.4: Could the authors further analyze how these failure modes arise during task execution, particularly how LLMs interpret structural objectives and coordinate constraints? Such analysis could help identify the specific reasoning limitations behind failures and inform targeted improvements.

  • Appendix B.11: Could the authors provide further insights into why different LLMs select different hotspots and diffusion regions? Comparing these decisions with the resulting structural outcomes could help distinguish errors in spatial planning from errors in constraint compilation, offering more actionable guidance for improving protein-editing agents.

  • More broadly, insights into how to improve the individual capabilities of LLMs and backend models, as well as their coordination, would make the work more impactful

5 comments
AuthorsOpenReview ↗

Response to Reviewer KaNV(part 1)

We sincerely thank the reviewer for the thorough and encouraging evaluation. Following your suggestions, we will incorporate your analytical framework to expand Section 4.4 and Appendix B.11 in the revised manuscript. Below, we address each weakness and question point by point.

W1:How can intermediate decision rationales and parameter choices be recorded to explain performance differences?

We fully adopted this advice and conducted a new diagnostic experiment across our evaluated models. We introduced an intermediate parameter-decision note to record structural rationales, target windows, spatial hotspots, and backend choices before producing the final configuration.

Implementation of structured decision ledgers:

Before generating the backend specification, the model records its reasoning across five explicit intermediate fields: (1) identified 3D structural elements, (2) designated sequence redesign windows, (3) continuous backbone diffusion scopes, (4) directional reference hotspots, and (5) justification for backend selection.

Key diagnostic insight:

Across over 97% of successfully completed decision logs, we found that models rarely fail due to an inability to format code. Instead, recording these intermediate decisions exposes the exact disconnect between biological intent and parameter assignment. In the majority of failed cases, models accurately recognize structural features in text, but fail to translate them into corresponding protection parameters.

We will formalize this parameter-decision ledger and documented the complete set of model decision records in Appendix B.13 of the revision.

In short, collecting intermediate parameter ledgers effectively clarifies why models diverge, proving that failures arise during parameter mapping rather than basic syntax generation.

AuthorsOpenReview ↗

Response to Reviewer KaNV(part 2)

Q1: How do failure modes arise during execution across structural understanding, spatial planning, and constraint coordination?

To analyze how failures arise during execution, we examine 20 evaluation cases across four frontier models (Qwen3.8-Max, DeepSeek-V4.1-Flash, Claude-Opus-5, and GPT-6-Sol; 80 total calls) alongside our main benchmark results (Section 4.4, Table 1). Failures emerge across three successive stages with distinct model-specific failure logics:

1. Structural Understanding & Rule Translation:

Models reliably identify coarse structural motifs, but falter on local residue identities and tool specifications (Section 4.4 reports intent-translation failure rates of 77.7%–97.7%). For example, in core burial, GPT-6-Sol identifies native disulfide bonds but misidentifies A148=ASN as Gly148, generating an erroneous "retain Gly148-Gly149" instruction. In cyclization, DeepSeek-V4.1-Flash misinterprets tool capabilities by claiming BoltzGen cannot support head-to-tail cyclization.

2. Spatial Planning:

Spatial planning represents a universal weakness across all evaluated models, failing consistently regardless of model family or backend architecture. This bottleneck stems from three specific geometric limitations:

  • Directional pulling vs. force cancellation: In inter-domain distance modulation, generative backends require a localized cluster of contact points to establish a coherent directional pulling vector. Models fail to plan this geometry: they either select dispersed contact points across the entire interface (causing opposing pulling vectors to cancel each other out) or command rigid translations without designing flexible hinge regions, transmitting excessive torque into the core (joint success drops to 12.0%–36.0%).

  • Local vs. non-local contact order: An \alpha-helix requires only local, contiguous (i \to i+4) hydrogen bonds, which generative backends nucleate readily from adjacent residues. In contrast, forming a \beta-sheet requires coordinating non-local strand pairing and geometric registry across 3D space. Models struggle to coordinate this non-local spatial topology, causing \beta-sheet success to lag 20–30 percentage points behind \alpha-helix across all models.

  • Lack of quantitative 3D calculation: Models cannot compute Euclidean coordinates accurately. In approach tasks, even Claude-Opus-5 selects candidate contact points that diverge from the true geometric nearest-point union (A137/A139/A140/A211/A219/A233 vs. true A137/A139/A140/A219/A232).

3. Constraint Coordination:

Multi-constraint coordination exhibits the highest failure rate (76.3%–92.3% in Section 4.4). In plain terms, models fall into a zero-sum dilemma: achieving the target structural change frequently tears apart the protected scaffold.

  • The cyclization deadlock as a clear example: If evaluated in isolation, models reach 100% on forming the head-to-tail peptide bond (G_1) and 100% on keeping the scaffold untouched (G_3). However, in 3D space, the N- and C-termini are separated by significant distance. Forcing them into bonding distance (1.2–1.5 Å) generates immense backbone tension. Because models fail to allocate flexible buffer windows to absorb this pull, closing the bond inevitably deforms the invariant scaffold beyond our 0.1 Å threshold. Consequently, the joint success rate completely collapses to 0.0%.

  • Across 6 cases containing native disulfide bonds, models handle this physical tension through distinct failure mechanisms:

    • DeepSeek-V4.1-Flash (Sacrifices Chemistry for Edit Goals): Protects 0/6 critical Cys residues. It mechanically releases the entire candidate window to downstream tools (e.g., mutating B96–107 despite a native 2.04 Å disulfide bond at B96–B22), completely ignoring covalent stability.

    • Qwen3.8-Max (Decision–Parameter Mismatch): Protects only 1/6 critical Cys residues. It frequently states that bonds must be preserved in its text reasoning, but forgets to declare the corresponding pinning parameters in the backend configuration.

    • GPT-6-Sol (Chemical Awareness with Interface Fragility): Protects 4/6 disulfide bonds (e.g., locking Cys109 and Gly113 while redesigning surrounding positions), but fails on interface-level schema formatting.

    • Claude-Opus-5 (Local Coordination vs. Global Strain): Protects 6/6 disulfide bonds locally, but still induces scaffold strain leakage during 3D generation, capping its joint G_1 \wedge G_3 rate at 46.0%.

We will update Section 4.4 to incorporate these stage-wise breakdowns and model-specific failure profiles.

AuthorsOpenReview ↗

Response to Reviewer KaNV(part 3)

Q2:Why do different LLMs select distinct hotspots and diffusion regions, and how do these decisions affect resulting structural outcomes?

We analyze why models make different hotspot and diffusion region choices, and we connect these parameter decisions directly to the resulting structural outcomes.

Q2.1: Why LLMs select different hotspots (Spatial planning vs. Structural outcome):

Concrete task context:

In an inter-domain separation task (appendix B.9), the goal is to move the editable core (chain B residues 67–74) away from a reference interface. The task card provides all 12 candidate interface residues on chain A: {62, 64, 95–97, 101–102, 141, 143, 146–148}.

Decision rationale across models:

Generative backends (such as ODesign) use the hotspot parameter to define the spatial direction of generation:

  • Claude-Opus-5 analyzes the 3D contact geometry and curates only 3 distal residues (A/62, A/141, A/147). This creates an asymmetric pulling vector pointing away from the core.

  • Kimi-K3 lacks directional spatial planning and mechanically copies all 12 candidate residues into its hotspot specification.

  • Qwen3.8Max over-prunes the list down to a single residue (A/62), which provides an insufficient steering signal.

Resulting structural outcomes:

  • Claude's curated 3-residue hotspot successfully steers the downstream backbone generator, achieving the required median distance increase (Δdist≥1.0 A˚) and passing the target gate (G1).

  • In contrast, Kimi's 12-residue hotspot envelops the entire interface. The competing pulling forces cancel each other out in 3D space, resulting in near-zero displacement and failing the target gate (G1).

  • This direct comparison proves that hotspot selection is not a syntax formatting issue, but a 3D spatial planning decision that dictates whether downstream generation succeeds.

Q2.2: Why LLMs select different diffusion regions (Constraint coordination vs. Structural outcome):

Concrete task context and physical strain:

In secondary-structure switching (β→α, residues A:217–224), the initial extended β-strand spans an end-to-end anchor distance of 28.5 Å (A:216 to A:225, verified as 28.5075 Å). Converting these 8 residues into a compact α-helix shortens the required backbone span by more than 10 Å, creating severe conformational strain against the fixed protein scaffold.

Decision rationale across models:

Models diverge in how they assign continuous 3D coordinate diffusion windows (partial_diff) versus discrete sequence redesign masks:

  • Claude-Opus-5 recognized this 28.5 Å span conflict and specifically selected ODesign. It restricted sequence mutations strictly to the core (A:217–224), but autonomously expanded the structural diffusion window (partial_diff) to include the flanking relaxation shell (A:212–229) to absorb backbone shortening.

  • Qwen3.8-Max and DeepSeek-V4.1-Flash selected backends (such as BoltzGen) where structural movement and sequence mutation are coupled to the exact same window.

  • In a separate approach task (Figure 9B, core B:99–110), weaker models (deepseek, Kimi, and Qwen) conflated the editable core with the relaxation shell and assigned partial_diff across the entire B:91–118 region.

Resulting structural outcomes:

  • In the β→α task, confining diffusion strictly to the 8-residue core without shell flexibility creates unresolvable steric clashes, causing downstream generation to distort bond geometry and fail ESMFold refolding (G2). Expanding the coordinate diffusion window to A:212–229 distributes the torsional strain across the shell, producing a stable helix.

  • Conversely, in Figure 9B, diffusing through the protected relaxation shell (B:91–118) allowed the generative backend to drift coordinates in the conserved scaffold, causing catastrophic scope boundary failure (G3).

  • This shows that diffusion window selection is a delicate trade-off: it must be broad enough to dissipate physical strain (avoiding G2 refolding failure), yet strictly bounded to prevent coordinate leakage into the invariant scaffold (avoiding G3 scope failure).

In summary, hotspot selection and diffusion scope assignment are not arbitrary formatting choices; they directly dictate physical outcomes. Hotspots provide the directional steering vector, while diffusion windows balance torsional strain dissipation against scaffold coordinate protection.

Reviewer_eTN9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PROTEAN, a benchmark for text-guided protein structure editing, comprising 380 tasks across eight categories. The framework uses LLMs to translate natural-language instructions into constraints for three protein generation backends and evaluates target attainment, scaffold preservation, and refolding consistency. The authors compare five LLMs and identify structural constraint violations and refolding failures as major bottlenecks.

Critical strengths
  1. Text-guided editing of existing protein structures is an interesting and practically important direction complementing the current emphasis on de novo protein generation.
  2. Separating target attainment, scaffold preservation, and refolding consistency is a useful evaluation framework for understanding the limitations of protein design pipelines
  3. The LLM-as-judge analysis of failure modes is an interesting attempt to understand model limitations beyond aggregate success rates.
Critical weaknesses
  1. Despite framing PROTEAN as a benchmark for text-guided protein editing, the LLM receives a highly structured task card containing the explicit action, residue masks, sequence constraints, and numerical objectives (Section B2–B3). One could decompose the benchmark into two parts: scientific capability of reasoning/planning protein editing given the user input and SWE capability of executing protein editing (using tools given the detailed action plans). PROTEAN largely evaluates the latter, which is less compelling.

  2. The benchmark primarily evaluates geometric changes with limited connection to functional protein engineering (despite this being the primary motivation of the paper), such as binding affinity or other funcitonal/fitness output. Geometric and refolding consistency do not necessarily imply functional improvement. It would be valuable to include function-oriented tasks, such as verifiers on binding affinity or fitness from deep mutational scanning (DMS, which could be a good test for single AA edits).

Minor

line 072: PROTEANinstitutes -> PROTEAN institutes

figure2: Feasi-bility check -> Feasibility

Questions for authors

See the weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer eTN9 (part 1)

W1:Does the benchmark evaluate genuine scientific reasoning or merely SWE execution of pre-determined action plans?

We thank the reviewer for this insightful perspective. We agree with the reviewer's two-part decomposition, however, in PROTEAN, we view scientific reasoning and SWE execution not as isolated capabilities, but as two sequential, indispensable steps in protein structural editing. Both parts are essential to the final outcome (i.e., whether a protein editing task succeeds or fails).

First, the model must perform scientific reasoning to analyze 3D geometry and physical constraints (see the example below). Second, the model uses this scientific understanding to write executable backend parameters. Pure SWE code generation cannot succeed without the preceding scientific reasoning.

Two-stage workflow in protein structural editing:

  1. Stage 1 (Scientific Planning): The model inspects local 3D coordinates, identifies native secondary structures, and assesses non-local spatial packing (such as salt bridges or steric hindrances) against the natural-language goal.

  2. Stage 2 (SWE Execution): The model maps the scientific conclusions from Stage 1 into the specific syntax of downstream generative backends (e.g., configuring partial_diff and sequence redesign masks in ODesign).

Direct empirical evidence of the two-stage process:

Our model outputs directly illustrate how scientific reasoning guides SWE execution across a complete three-step workflow (e.g., on the alpha-helix enrichment task, residues A:295–302), which is also enclosed in our latest version of paper:

Step 1: 3D geometric and secondary-structure analysis (Scientific Planning)

The model reads backbone dihedrals and hydrogen-bond distances to diagnose the current fold:

"All context residues A:291-A:306 display canonical alpha-helical phi/psi angles clustered tightly around (-60°, -45°)... The short_backbone_O_to_N_pairs table reveals a continuous i→i+4 hydrogen-bond ladder characteristic of an alpha-helix: A:291:O→A:295:N (2.88 Å), A:292:O→A:296:N (2.93 Å)... A:295:O→A:299:N (2.95 Å)... Core backbone atoms for A:295-A:302 are fully resolved." (Independent DSSP analysis confirms that all eight core residues adopt the alpha-helix state H).

Step 2: Tertiary packing and biophysical constraints evaluation (Scientific Planning)

The model identifies non-local contacts that restrict mutations:

"The core is heavily packed against non-local context residues: H296 contacts Q83/R84/H85/W86; W298 packs against L264/K265/F266/F271; K299 NZ forms a close contact with E88 OE1 (2.65 Å). This dense tertiary packing constrains allowable sequence substitutions."

Step 3: Downstream tool configuration (SWE Execution)

The model uses the conclusions from Steps 1 and 2 to configure the ODesign backend:

"ODesign is selected because it exposes a partial_diff field that localizes structural diffusion precisely to the edit core, which is the most direct available mechanism for encouraging an alpha-rich conformational shift."

"Because the target criterion is alpha_rich... and the observed fold is already a stable alpha-helix, the primary goal is to preserve this geometry while optimizing the sequence for intrinsic helical propensity. Redesigning exactly the 8-residue edit core (295-302) allows the model to substitute G301 and C300 with stronger helix formers... while keeping the flanking shell fixed... Setting partial_diff over the core enables minor backbone relaxation... without disrupting the established i→i+4 H-bond network."

The model then outputs the exact backend parameters: "sequence": "A/1-294,8-8,A/303-597", "partial_diff": "A/295-302".

The task card provides only the physical boundaries of the problem (analogous to spatial masks in image inpainting). It does not provide mutation choices, spatial displacement values, or tool-specific parameters. A simple SWE parser that lacks scientific reasoning would not know which residues can be mutated, whether to relax the backbone, or how to set the diffusion window.

Overall, PROTEAN evaluates the full continuum from biophysical reasoning to software execution. Scientific planning dictates what physical edits are necessary, and SWE execution translates those requirements into machine-executable parameters for generative backends.

AuthorsOpenReview ↗

Response to Reviewer eTN9 (part 2)

W2:Does geometric and refolding compliance translate to biological function, and can function-oriented tasks (e.g., DMS fitness) be evaluated?

We thank the reviewer for this constructive suggestion. We agree that verifying real experimental endpoints is important for protein engineering. To evaluate functional fitness, we provide an additional function-oriented editing task:

Task definition: Given (1) a starting protein variant, (2) an allowed editable window, and (3) a functional goal in natural language, the model must propose an amino-acid substitution to improve biological fitness.

Datasets and functional assays:

We adapted experimental assays across three biophysical functions:

  • GFP fluorescence restoration (brightness assay).
  • PDZ3 domain–peptide binding recovery (binding enrichment assay).
  • YAP WW domain binding compensation (double-mutant binding assay).

Performance trends against experimental DMS records:

Model outputs were evaluated directly against real experimental measurements:

  • Qwen3.8-Max: consistently generated valid mutation actions, and the vast majority of its proposed mutations successfully improved experimental fitness scores.

  • DeepSeek-V4.1-Flash: reliably produced valid mutation actions, with most proposed substitutions achieving higher experimental scores than the starting variants.

  • Claude-Opus-5: all of its completed mutation actions successfully improved experimental scores, frequently selecting the top-scoring candidate within the experimental library.

Discrimination of model decisions (PDZ3 binding recovery):

Models made different mutation choices under the exact same boundary mask, leading to different experimental outcomes:

  • On the L323S variant of the PDZ3 domain (starting binding score: −0.2521), Claude chose S323F to restore hydrophobic core packing. The experimental binding score improved to −0.01445.

  • Qwen and DeepSeek chose S323A. The experimental binding score for S323A was −0.25676, which was slightly lower than the starting variant.

Representative mutation choices across other assays:

  • In GFP fluorescence recovery (F100S background, starting brightness: 1.9135), all three models selected S100Y. The experimental brightness increased to 3.5917.

  • In YAP WW binding compensation (E178N background, baseline score: 0.6153), Qwen and DeepSeek selected K181R. The experimental binding score increased to 0.8864.

This pilot test evaluates single-site functional actions against fixed experimental libraries. We will provide the full dataset curation details, task cards, and experimental score tables in Appendix B.12 of the revision.

Due to the rebuttal time limit, we directly adapted experimental datasets from ProteinGym, and we will include expanded evaluations and more data in the revision.

In summary, this experiment demonstrates that our benchmark pipeline can quickly adapt to new function-oriented tasks. By connecting model actions to wet-lab DMS scores, we found a valuable phenomenon: frontier models can effectively perceive functional fitness landscapes from natural-language goals and propose mutations that directly rescue experimental activity.

AuthorsOpenReview ↗

Follow up to Reviewer eTN9

Thank you again for your valuable review and helpful suggestions. As the discussion deadline is approaching, we wanted to kindly follow up regarding our earlier responses and the revised manuscript.

In response to your feedback, we provided a three-step reasoning trace illustrating how scientific planning guides SWE execution (added in Appendix B.13) and conducted a new function-oriented DMS pilot across GFP, PDZ3, and YAP WW assays (added in Appendix B.12).

We hope our responses and revisions have addressed your concerns. Please let us know if you have any remaining questions—we are very open to further discussion.

Reviewer_dLLeOpenReview ↗
Rating
3(+2)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PROTEAN, a benchmark for text-guided 3D protein structure editing. PROTEAN targets localized structural remodeling while preserving unedited regions in both sequence identity and 3D backbone coordinates. The benchmark comprises 380 tasks across six families, including 350 length-preserving in-scaffold editing tasks and 30 variable-length motif-scaffolding tasks. It use upstream LLM compiles task instructions into backend-specific configuration parameters, and a downstream 3D generative model to execute the protein remodeling.

Critical strengths
  1. The paper proposed a novel task on text-guided protein design. Rather than utilize LLM to generate protein by itself, it test the LLM capability in protein editing with speicalized protein design model.
  2. The dataset curation process is well-motivated and inspiring.
Critical weaknesses

There are a lot of serious issues with this paper and make me confusing. I could be wrong about these and I hope the authors can correct me:

  1. The first biggest issue of this paper is, more than half of the tasks constructed by author, the input text already contain all the information needed in the output json. All the regions and range are specified in the task card, what LLM need to do is parsing them out. For this type of task, the authors are basically testing modern LLM parsing capability. It looks like there are only several real tasks that are testing LLM's capability about protein understanding, for example, the hotspot selection. In CASE 1 — SEPARATION: HOTSPOT CURATION, the authors show that "Claude-Opus-5 intentionally curates 3 of the 12 candidate patch residues" and claims that "encodes an authentic directional comprehension of interface mechanics." And that brings the second biggest issue:
  2. The authors repeatedly claim something like "Together, while our findings establish that frontier LLMs possess genuine 3D spatial planning competencies". And at the very beginning, the authors define the source protein in PROTEAN has S (sequence) and X (strucutre, coords of four atoms). So I assume that the protein structure is used as the input to the LLM. However, in B.3 WHAT THE MODEL SEES, the authors said each attempt is one single API call. The example the author gave has 12,462 characters. In B.2, the authors show what a task card looks like and the source is ommited ("source": {...}). This place should be the source protein sequence and structure. But how can they input the protein structure given that the whole prompt is only 12,462 characters? The authors made a lot of claims that LLM have 3D understanding of protein, but the 3D structure are unlikely use as input here.
  3. The color in the input prompt also feels so weird, what's the point of telling LLM "red" or "purple" region? I didn't see the authors specify any color in the task card.
  4. In Table 1, Localized structural transplantation & repacking task, for Qwen3.8Max under Fragment mimicry, the G1 is 18 and G3 is 78, how is G1&G3 22?
  5. In Table1, all the LLM share the same G3 scores?
Questions for authors

How are protein structures used as input to the LLM? If they are not used as input, how did the authors claim about LLM's 3D spatial planning competencies?

15 comments
AuthorsOpenReview ↗

Response to Reviewer dLLe (part 1)

Thank you for your detailed review comments. Below is our response.

W1: Reviewer concern:

"In more than half of the tasks ... The input text already contains all the information needed in the output JSON ... testing modern LLM parsing capability with only several real tasks testing LLM's capability for protein understanding..."

W1.1:Does the input task card already contain all the information needed in the solution?

We clarify that the input task card does NOT contain the specific solutions required in the output. The input contains only three essential components: (1) the editing goal and evaluation metric (target_criteria), (2) region boundaries and sequence policies (roles, sequence_policy), and (3) the initial 3D structural context (source). These components are all necessary physical conditions to define the engineering problem.

The evaluation criteria (target_criteria) define the quantitative goal and passing thresholds (e.g., reducing distance by at least 0.5 Å). They state what physical change must be achieved, without specifying any backend parameters or geometric strategies to reach it.

Region boundaries and sequence rules (roles, sequence_policy) act as physical boundary constraints. In real-world protein engineering, researchers rarely build proteins from scratch; they modify existing natural scaffolds to preserve stability and solubility [1]. Specifying roles marks the editable core and the protected scaffold, preventing the task from degenerating into unconstrained de novo design. Similarly, sequence_policy (such as preserving sequence length) prevents shortcuts like deleting residues to avoid clashes. These rules define where edits are legal, not how to solve the edit.

The initial 3D structural context (source) provides the physical starting state of the protein. The model must analyze the preprocessed 3D features in source (such as backbone dihedrals and spatial contact tables from Gemmi) to deduce downstream parameters autonomously. For example, in secondary-structure switching on residues A:217–224 (an 8-amino-acid segment), the input DOES NOT contain the following essential output parameters:

  1. Tool selection: The task card never specifies which downstream backend to use (e.g., BoltzGen vs. ODesign).

  2. Coordinate diffusion ranges: The model autonomously expands the continuous coordinate diffusion window (partial_diff) to A:212–229 to absorb a 28.5 Å backbone span conflict.

In short, the task card defines the physical boundaries required by real-world protein engineering, while all generative control parameters must be actively derived through 3D structural reasoning.

[1] Romero et al., Exploring protein fitness landscapes by directed evolution, Nature Reviews Molecular Cell Biology 2009.

W1.2: The concern that the benchmark only evaluates modern LLMs' text parsing capability.

PROTEAN does not evaluate simple text parsing, but rather biophysical reasoning under coupled physical constraints. Naively copying input boundaries into backend specifications leads to complete failure under our metrics.

PROTEAN evaluates real physical structure constraints: achieving target modifications while strictly keeping scaffold displacement under 0.1 Å (G3). A simple text-parsing strategy completely fails under this physical threshold. In our controlled test, directly converting input mask indices into backend parameters without physical reasoning resulted in a 0% pass rate (0/50 samples succeeded).

To pass the benchmark gates, the framework forces models to perform real biophysical reasoning. For example, in a core burial task on candidate window A:106–113:

  • The model identifies a native disulfide bond between Cys109 and Cys210 (2.081 Å sulfur–sulfur contact).

  • To preserve scaffold stability while reducing solvent accessibility, the model actively preserves Cys109 instead of mutating the whole window.

  • It autonomously partitions the edit scope into two separate redesign segments (A:106–108 and A:110–113).

This shows that the model resolves physical conflicts rather than performing text extraction. We provide a complete, step-by-step reasoning workflow from a diagnostic probe in W1.3 below.

AuthorsOpenReview ↗

Response to Reviewer dLLe (part 2)

W1.3:The concern that most tasks fail to test LLMs' protein understanding.

Protein structural understanding is required across all eight editing tasks, not just in hotspot selection. To verify whether parameter outputs reflect authentic structural understanding across different task families, we conducted a diagnostic probe under identical single-call inputs. By allowing the model (Qwen3.8-Max) to explicitly record its structural assessment fields, we observe a complete, authentic three-step biophysical reasoning workflow: diagnosing 3D backbone dihedrals and hydrogen-bond networks, identifying non-local packing constraints, and configuring the downstream generative tool (demonstrated below on the α-helix enrichment task, residues A:295–302):

Step 1: 3D geometric and secondary-structure analysis The model reads backbone dihedrals and hydrogen-bond distances to diagnose the current fold:

"All context residues A:291-A:306 display canonical alpha-helical phi/psi angles clustered tightly around (-60°, -45°)... The short_backbone_O_to_N_pairs table reveals a continuous i→i+4 hydrogen-bond ladder characteristic of an alpha-helix: A:291:O→A:295:N (2.88 Å), A:292:O→A:296:N (2.93 Å)... A:295:O→A:299:N (2.95 Å)... Core backbone atoms for A:295-A:302 are fully resolved." (Independent DSSP analysis confirms that all eight core residues adopt the alpha-helix state H).

Step 2: Tertiary packing and biophysical constraints evaluation The model identifies non-local contacts that restrict mutations:

"The core is heavily packed against non-local context residues: H296 contacts Q83/R84/H85/W86; W298 packs against L264/K265/F266/F271; K299 NZ forms a close contact with E88 OE1 (2.65 Å). This dense tertiary packing constrains allowable sequence substitutions."

Step 3: Downstream tool selection and configuration The model translates these biophysical observations into exact parameters for the ODesign backend:

"ODesign is selected because it exposes a partial_diff field that localizes structural diffusion precisely to the edit core, which is the most direct available mechanism for encouraging an alpha-rich conformational shift."

"Because the target criterion is alpha_rich... and the observed fold is already a stable alpha-helix, the primary goal is to preserve this geometry while optimizing the sequence for intrinsic helical propensity. Redesigning exactly the 8-residue edit core (295-302) allows the model to substitute G301 and C300 with stronger helix formers... while keeping the flanking shell fixed... Setting partial_diff over the core enables minor backbone relaxation... without disrupting the established i→i+4 H-bond network."

The model then outputs the exact backend JSON: "sequence": "A/1-294,8-8,A/303-597", "partial_diff": "A/295-302". The Chain of Thought by Qwen3.8-Max is also enclosed in our latest version of paper.

In summary, our benchmark evaluates whether models can turn high-level language goals into valid structural parameters. Both the quantitative benchmark scores and the diagnostic probe confirm that LLMs reason over physical 3D constraints during execution.

W2:Clarification on 3D structure input representation under character limits.

The 12,462-character count in Appendix B.3 represents only the base prompt skeleton where the source structure was abbreviated. In actual execution, the Gemmi-preprocessed structural information adds an average of approximately 8,695 characters per sample. The model directly receives these compact 3D structural features in the task card, and it performs 3D spatial planning by converting these constraints into downstream generative parameters.

Structure input via Gemmi preprocessing: We use Gemmi to parse raw CIF files during data preprocessing. Gemmi extracts sequences, chains, residue boundaries, and 3D spatial contact pairs into structured text. This information is provided to the model in the task card. The model reads these 3D features directly from the input prompt.

Specific workflow of 3D spatial planning: 3D spatial planning means the model analyzes these spatial constraints and outputs precise parameters for downstream backends. As shown in appendix B.9:

  • In separation tasks, the model selects a specific spatial cluster of residues as hotspots to create an effective directional pulling vector.
  • In approach tasks, the model separates the editable core from the relaxation shell to protect invariant scaffold coordinates.
  • In the revision, we will add explicit details about the Gemmi preprocessing pipeline and provide the full JSON schema of the preprocessed 3D input in the appendix.

In short, the model ingests pre-parsed 3D structural data from the task card and accomplishes 3D spatial planning through end-to-end parameter compilation.

AuthorsOpenReview ↗

Response to Reviewer dLLe (part 3)

W3: What is the purpose of visual color terms (e.g., "red", "purple") in the input instructions?

We would like to clarify that the color terms and instruction labels in Figure 1 and Table 4 are purely for visual illustration for readers (i.e., human). They are not part of the actual execution workflow for the model. In Figure 1, for example, red is used to highlight the alpha-helix to be edited, and purple denotes the reference landmark. These visual colors are not part of the benchmark's actual execution process. It is not the raw input prompt received by the model. To prevent misunderstandings, we have replaced the label "Instruction" in Figure 1 with "Editing Objective" in the revision.

We outline the overall workflow as follows:

  • The model receives the structured task card. Physical target regions are strictly governed by the machine-readable roles dictionary rather than color names.

  • For example, in the instruction "Move the red edit core closer to the purple reference":

    • roles.edit_core defines chain B residues 99–110.
    • roles.reference_patch defines chain A residues 7–12, 193, 196–197, and 201.
    • The model maps these functional roles to the physical residues and does not interpret visual colors.

In short, the colors and labels in Figure 1 and Table 4 are visual aids for readers. The model operates entirely on structured residue indices in the task card. We have updated the labels in Figure 1 and Table 4 to make this distinction completely clear.

W4 & W5: Corrections regarding Qwen's arithmetic score in Fragment Mimicry and identical G3 columns across models in Table 1.

We sincerely apologize for the inadvertent typographical and transcription errors in Table 1. In the submitted manuscript, Qwen3.8Max's G_1 under Fragment mimicry was miswritten as 18 instead of its actual value of 28. Additionally, the G_3 column was duplicated across models due to a consolidation script error that referenced a shared baseline slice.

We have carefully verified the original logs and updated Table 1 in Section 4.2 (Page 7) as follows:

Task / FamilyDeepSeek-V4-Pro(G_1 / G_3 / ∧)Qwen3.8Max(G_1 / G_3 / ∧)Claude-Opus-5(G_1 / G_3 / ∧)GPT-5.6-Sol(G_1 / G_3 / ∧)Kimi-K3(G_1 / G_3 / ∧)
Secondary-structure
α\alpha-Helix86 / 92 / 66.088 / 90 / 58.098 / 92 / 90.096 / 94 / 84.088 / 82 / 76.0
β\beta-Sheet62 / 80 / 44.060 / 82 / 36.084 / 84 / 60.082 / 86 / 58.060 / 76 / 40.0
Distance modulation
Approach36 / 72 / 18.038 / 74 / 12.056 / 78 / 36.046 / 66 / 26.048 / 52 / 14.0
Separation46 / 82 / 20.044 / 68 / 12.048 / 80 / 34.052 / 50 / 28.050 / 50 / 18.0
Transplantation & burial
Fragment mimicry18 / 76 / 16.028 / 78 / 22.076 / 78 / 56.058 / 78 / 38.018 / 74 / 14.0
Core burial34 / 48 / 30.036 / 58 / 28.066 / 68 / 46.050 / 70 / 34.034 / 52 / 20.0
Topological alteration
Cyclization100 / 100 / 0.0100 / 100 / 0.0100 / 100 / 0.0100 / 100 / 0.0100 / 100 / 0.0
Overall54.6 / 78.6 / 27.756.3 / 78.6 / 24.075.4 / 82.9 / 46.069.1 / 77.7 / 38.356.9 / 69.4 / 26.0

Response to Details of Ethics Concerns: Clarification and Correction of Bibliography Duplications and Formatting Artifacts.

We thank the reviewer for carefully pointing out these citation duplications and name discrepancies.

All cited papers were manually searched and selected by the authors ourselves. Every referenced paper is an authentic published work in the field. The duplicate entries arose because different co-authors independently added the same references during collaborative Overleaf compilation.

We would like to clarify that the references are not generated by the LLMs, as the papers do in fact exist. The name and venue discrepancies occurred during AI-assisted text polishing (as declared in our AI Use Statement): Publication venue for ProteinDT: The paper is published in Nature Machine Intelligence (2025). The tool mistakenly retained an outdated draft tag alongside the correct citation. Author names: The formatting tool inadvertently altered several author first names during automated reference reformatting. In the latest version of paper, we have removed all duplicates, correct all author names and venues against official publisher records, and manually check the entire bibliography.

We sincerely thank the reviewer for the careful reading, valuable comments, and constructive suggestions, which have greatly helped us improve the quality and rigor of this manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We thank all reviewers for their careful reading and constructive feedback. We have uploaded the revised manuscript incorporating all requested corrections, analyses, and documentation updates. Below, we briefly summarize our responses to the shared questions across reviews and list all revisions.

Summary of Shared Questions

  1. Biophysical reasoning vs. text parsing / SWE execution (Reviewers eTN9, dLLe, KaNV): The task card specifies physical boundaries (roles) and goals (target_criteria), not backend solutions. A mechanical rule-based copying baseline fails (0/50). Models must analyze 3D geometry and tertiary packing to select backends, diffusion spans (partial_diff), and hotspot residues. On the exact same input, different models reason differently and produce different parameters (Appendix B.13, Figure 9).
  2. 3D structure input and mechanistic failure analysis (Reviewers dLLe, KaNV): During preprocessing, MMseqs2, gemmi, numpy, scipy, mkdssp, and Shrake–Rupley extract compact 3D geometric features (backbone coordinates, dihedrals, contacts, and distances) into the task card, while the harness passes canonical structure.cif directly to each 3D backend (Appendix B.3). Our intermediate decision-ledger and 80-call diagnostic analyses show that model failures stem primarily from 3D spatial planning and multi-constraint coordination rather than JSON formatting.
  3. Connection to biological function (Reviewer eTN9): We added a function-oriented editing pilot evaluated against experimental DMS measurements from ProteinGym across GFP, PDZ3, and YAP WW assays (Appendix B.12, Table 7). Under identical editable masks, models propose distinct substitutions that lead to different measured fitness outcomes.

Summary of Manuscript Revisions

  1. Figure 1 & Table 4 (Pages 3, 17): Updated labels from "Instruction" to "Editing Objective" and fixed minor typos (e.g., Page 2, Line 073; Page 23 table reference).
  2. Table 1 (Page 7): Corrected Qwen3.8Max's Fragment Mimicry G1G_1 score (18 → 28) and updated the G3G_3 columns with verified per-model values from raw logs.
  3. References (Pages 10–13): Removed duplicate entries and verified all author names and venues against official records (including MotifBench by Zhuoqi Zheng et al., 2025, and ProteinDT by Shengchao Liu et al., Nature Machine Intelligence, 2025).
  4. Appendix B.3 (Pages 16–19): Updated the system prompt to reflect autonomous multi-backend routing and added the step-by-step 3D preprocessing pipeline (gemmi, biotite, numpy, scipy, PyYAML, and json/hashlib).
  5. Appendix B.12 (Pages 27–28, Table 7): Added the ProteinGym DMS functional-editing pilot.
  6. Appendix B.13 (Page 29): Added the three-step reasoning trace (A:295–302) linking 3D scientific planning to downstream backend configuration.

We sincerely thank Reviewer KaNV and Reviewer eTN9 for their supportive assessments and constructive suggestions, and Reviewer dLLe for the detailed technical scrutiny that helped us fix table, citation, and documentation issues. We will release a clean codebase and full prompt files for reproducibility.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.