- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
-
PROTEAN addresses the problem of targeted protein structure editing, where specific regions need to be modified while preserving functional motifs and the surrounding structural scaffold. It proposes a two-stage pipeline in which an LLM translates natural-language instructions into backend-specific JSON specifications, which are then executed by protein-design models. The authors introduce a benchmark of 380 tasks covering eight types of structural editing and evaluate five LLMs across three backends: BoltzGen, ODesign, and RFdiffusion3.
-
Experiments show that Claude-Opus-5 achieves the highest joint geometric compliance of 46.0% under pass@16 for length-preserving editing tasks. The analysis identifies structural scope leakage, spatial-planning errors, and refolding failures as major bottlenecks. Although the framework demonstrates the potential of using LLMs to orchestrate specialized protein-design models, the results also reveal substantial differences in how models translate structural objectives into backend parameters. A more detailed comparison of their decision-making processes could further clarify why some models produce more effective editing specifications than others.
Critical strengths
-
PROTEAN moves beyond de novo protein generation to address targeted editing of existing protein structures. Its benchmark comprises 380 tasks across eight editing types, with explicit constraints on editable regions and structural preservation.
-
By evaluating five LLMs across three generative backends, the study distinguishes geometric compliance from biophysical realizability, providing a more comprehensive assessment of whether generated structures satisfy editing requirements and remain physically plausible.
-
Detailed analyses of hotspot selection and diffusion-region specification reveal differences in how LLMs translate structural instructions into backend parameters. The findings highlight scope leakage and multi-constraint coordination as important challenges for future protein-editing agents.
Critical weaknesses
The paper does not sufficiently analyze the reasoning behind LLMs’ failure cases and backend parameter choices, limiting the understanding of why models differ in performance (protein-structure understanding, spatial planning, constraint coordination, or the interpretation of backend-specific parameters) and how their capabilities can be improved.
I understand that obtaining explicit reasoning traces from closed-source LLMs may be difficult. Nevertheless, the authors could consider alternatives, such as collecting structured decision rationales or recording intermediate parameter-selection decisions.
Questions for authors
-
Section 4.4: Could the authors further analyze how these failure modes arise during task execution, particularly how LLMs interpret structural objectives and coordinate constraints? Such analysis could help identify the specific reasoning limitations behind failures and inform targeted improvements.
-
Appendix B.11: Could the authors provide further insights into why different LLMs select different hotspots and diffusion regions? Comparing these decisions with the resulting structural outcomes could help distinguish errors in spatial planning from errors in constraint compilation, offering more actionable guidance for improving protein-editing agents.
-
More broadly, insights into how to improve the individual capabilities of LLMs and backend models, as well as their coordination, would make the work more impactful
5 comments
Response to Reviewer KaNV(part 2)
Q1: How do failure modes arise during execution across structural understanding, spatial planning, and constraint coordination?
To analyze how failures arise during execution, we examine 20 evaluation cases across four frontier models (Qwen3.8-Max, DeepSeek-V4.1-Flash, Claude-Opus-5, and GPT-6-Sol; 80 total calls) alongside our main benchmark results (Section 4.4, Table 1). Failures emerge across three successive stages with distinct model-specific failure logics:
1. Structural Understanding & Rule Translation:
Models reliably identify coarse structural motifs, but falter on local residue identities and tool specifications (Section 4.4 reports intent-translation failure rates of 77.7%–97.7%). For example, in core burial, GPT-6-Sol identifies native disulfide bonds but misidentifies A148=ASN as Gly148, generating an erroneous "retain Gly148-Gly149" instruction. In cyclization, DeepSeek-V4.1-Flash misinterprets tool capabilities by claiming BoltzGen cannot support head-to-tail cyclization.
2. Spatial Planning:
Spatial planning represents a universal weakness across all evaluated models, failing consistently regardless of model family or backend architecture. This bottleneck stems from three specific geometric limitations:
-
Directional pulling vs. force cancellation: In inter-domain distance modulation, generative backends require a localized cluster of contact points to establish a coherent directional pulling vector. Models fail to plan this geometry: they either select dispersed contact points across the entire interface (causing opposing pulling vectors to cancel each other out) or command rigid translations without designing flexible hinge regions, transmitting excessive torque into the core (joint success drops to 12.0%–36.0%).
-
Local vs. non-local contact order: An \alpha-helix requires only local, contiguous (i \to i+4) hydrogen bonds, which generative backends nucleate readily from adjacent residues. In contrast, forming a \beta-sheet requires coordinating non-local strand pairing and geometric registry across 3D space. Models struggle to coordinate this non-local spatial topology, causing \beta-sheet success to lag 20–30 percentage points behind \alpha-helix across all models.
-
Lack of quantitative 3D calculation: Models cannot compute Euclidean coordinates accurately. In approach tasks, even Claude-Opus-5 selects candidate contact points that diverge from the true geometric nearest-point union (A137/A139/A140/A211/A219/A233 vs. true A137/A139/A140/A219/A232).
3. Constraint Coordination:
Multi-constraint coordination exhibits the highest failure rate (76.3%–92.3% in Section 4.4). In plain terms, models fall into a zero-sum dilemma: achieving the target structural change frequently tears apart the protected scaffold.
-
The cyclization deadlock as a clear example: If evaluated in isolation, models reach 100% on forming the head-to-tail peptide bond (G_1) and 100% on keeping the scaffold untouched (G_3). However, in 3D space, the N- and C-termini are separated by significant distance. Forcing them into bonding distance (1.2–1.5 Å) generates immense backbone tension. Because models fail to allocate flexible buffer windows to absorb this pull, closing the bond inevitably deforms the invariant scaffold beyond our 0.1 Å threshold. Consequently, the joint success rate completely collapses to 0.0%.
-
Across 6 cases containing native disulfide bonds, models handle this physical tension through distinct failure mechanisms:
-
DeepSeek-V4.1-Flash (Sacrifices Chemistry for Edit Goals): Protects 0/6 critical Cys residues. It mechanically releases the entire candidate window to downstream tools (e.g., mutating B96–107 despite a native 2.04 Å disulfide bond at B96–B22), completely ignoring covalent stability.
-
Qwen3.8-Max (Decision–Parameter Mismatch): Protects only 1/6 critical Cys residues. It frequently states that bonds must be preserved in its text reasoning, but forgets to declare the corresponding pinning parameters in the backend configuration.
-
GPT-6-Sol (Chemical Awareness with Interface Fragility): Protects 4/6 disulfide bonds (e.g., locking Cys109 and Gly113 while redesigning surrounding positions), but fails on interface-level schema formatting.
-
Claude-Opus-5 (Local Coordination vs. Global Strain): Protects 6/6 disulfide bonds locally, but still induces scaffold strain leakage during 3D generation, capping its joint G_1 \wedge G_3 rate at 46.0%.
-
We will update Section 4.4 to incorporate these stage-wise breakdowns and model-specific failure profiles.
Response to Reviewer KaNV(part 3)
Q2:Why do different LLMs select distinct hotspots and diffusion regions, and how do these decisions affect resulting structural outcomes?
We analyze why models make different hotspot and diffusion region choices, and we connect these parameter decisions directly to the resulting structural outcomes.
Q2.1: Why LLMs select different hotspots (Spatial planning vs. Structural outcome):
Concrete task context:
In an inter-domain separation task (appendix B.9), the goal is to move the editable core (chain B residues 67–74) away from a reference interface. The task card provides all 12 candidate interface residues on chain A: {62, 64, 95–97, 101–102, 141, 143, 146–148}.
Decision rationale across models:
Generative backends (such as ODesign) use the hotspot parameter to define the spatial direction of generation:
-
Claude-Opus-5 analyzes the 3D contact geometry and curates only 3 distal residues (A/62, A/141, A/147). This creates an asymmetric pulling vector pointing away from the core.
-
Kimi-K3 lacks directional spatial planning and mechanically copies all 12 candidate residues into its hotspot specification.
-
Qwen3.8Max over-prunes the list down to a single residue (A/62), which provides an insufficient steering signal.
Resulting structural outcomes:
-
Claude's curated 3-residue hotspot successfully steers the downstream backbone generator, achieving the required median distance increase (Δdist≥1.0 A˚) and passing the target gate (G1).
-
In contrast, Kimi's 12-residue hotspot envelops the entire interface. The competing pulling forces cancel each other out in 3D space, resulting in near-zero displacement and failing the target gate (G1).
-
This direct comparison proves that hotspot selection is not a syntax formatting issue, but a 3D spatial planning decision that dictates whether downstream generation succeeds.
Q2.2: Why LLMs select different diffusion regions (Constraint coordination vs. Structural outcome):
Concrete task context and physical strain:
In secondary-structure switching (β→α, residues A:217–224), the initial extended β-strand spans an end-to-end anchor distance of 28.5 Å (A:216 to A:225, verified as 28.5075 Å). Converting these 8 residues into a compact α-helix shortens the required backbone span by more than 10 Å, creating severe conformational strain against the fixed protein scaffold.
Decision rationale across models:
Models diverge in how they assign continuous 3D coordinate diffusion windows (partial_diff) versus discrete sequence redesign masks:
-
Claude-Opus-5 recognized this 28.5 Å span conflict and specifically selected ODesign. It restricted sequence mutations strictly to the core (A:217–224), but autonomously expanded the structural diffusion window (partial_diff) to include the flanking relaxation shell (A:212–229) to absorb backbone shortening.
-
Qwen3.8-Max and DeepSeek-V4.1-Flash selected backends (such as BoltzGen) where structural movement and sequence mutation are coupled to the exact same window.
-
In a separate approach task (Figure 9B, core B:99–110), weaker models (deepseek, Kimi, and Qwen) conflated the editable core with the relaxation shell and assigned partial_diff across the entire B:91–118 region.
Resulting structural outcomes:
-
In the β→α task, confining diffusion strictly to the 8-residue core without shell flexibility creates unresolvable steric clashes, causing downstream generation to distort bond geometry and fail ESMFold refolding (G2). Expanding the coordinate diffusion window to A:212–229 distributes the torsional strain across the shell, producing a stable helix.
-
Conversely, in Figure 9B, diffusing through the protected relaxation shell (B:91–118) allowed the generative backend to drift coordinates in the conserved scaffold, causing catastrophic scope boundary failure (G3).
-
This shows that diffusion window selection is a delicate trade-off: it must be broad enough to dissipate physical strain (avoiding G2 refolding failure), yet strictly bounded to prevent coordinate leakage into the invariant scaffold (avoiding G3 scope failure).
In summary, hotspot selection and diffusion scope assignment are not arbitrary formatting choices; they directly dictate physical outcomes. Hotspots provide the directional steering vector, while diffusion windows balance torsional strain dissipation against scaffold coordinate protection.
Response to Reviewer KaNV(part 1)
We sincerely thank the reviewer for the thorough and encouraging evaluation. Following your suggestions, we will incorporate your analytical framework to expand Section 4.4 and Appendix B.11 in the revised manuscript. Below, we address each weakness and question point by point.
W1:How can intermediate decision rationales and parameter choices be recorded to explain performance differences?
We fully adopted this advice and conducted a new diagnostic experiment across our evaluated models. We introduced an intermediate parameter-decision note to record structural rationales, target windows, spatial hotspots, and backend choices before producing the final configuration.
Implementation of structured decision ledgers:
Before generating the backend specification, the model records its reasoning across five explicit intermediate fields: (1) identified 3D structural elements, (2) designated sequence redesign windows, (3) continuous backbone diffusion scopes, (4) directional reference hotspots, and (5) justification for backend selection.
Key diagnostic insight:
Across over 97% of successfully completed decision logs, we found that models rarely fail due to an inability to format code. Instead, recording these intermediate decisions exposes the exact disconnect between biological intent and parameter assignment. In the majority of failed cases, models accurately recognize structural features in text, but fail to translate them into corresponding protection parameters.
We will formalize this parameter-decision ledger and documented the complete set of model decision records in Appendix B.13 of the revision.
In short, collecting intermediate parameter ledgers effectively clarifies why models diverge, proving that failures arise during parameter mapping rather than basic syntax generation.