- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces a benchmark of 30 "hazardous" single-amino-acid mutation-design scenarios in 8 risk categories. The scenarios are filtered to proteins with deep mutational scanning data in ProteinGym/ViroGym. An LLM agent receives a wild-type sequence and an objective. It proposes candidates, optionally queries six biological design tools (MMseqs2 conservation, ESM-1v, ProteinMPNN, UniProt annotations, ToxinPred3, and a scenario-specific predictor: AlphaMissense / EVEscape / FoldX), and submits a top-3. Submissions are scored by looking up their measured DMS phenotype.
Critical strengths
- Scoring open-ended LLM designs against existing hidden DMS measurements avoids both LLM-as-judge and hazardous wet-lab validation.
- Careful attention to experimental hygiene in places. S0/S1/S2 share one initial pool C. Coverage of submissions by measurements is reported.The authors state plainly which mechanism controls were not run (App. D).
- Metric robustness checks. BRS vs. rank-based metric agreement and refusal-floor sensitivity are useful.
Critical weaknesses
-
Missing tool-only/human baseline yet the paper claimsLLMs and BDTs pose "substantial" risk. A cheap baseline model is missing: rank all single substitutions with the provided tools themselves (e.g. ESM-1v) and submit the top-3. These are the standard zero-shot ProteinGym baselines.
-
Memorization / contamination is only partly addressed. ProteinGym and most of the underlying DMS papers are public and likely in pretraining data. App. F.2 shows the top designs are often non-consensus, but that does not rule out recall of the DMS results themselves
-
Only single mutations are considered and this is likely not the actual case where BDTs gonna be used.
-
Authors could consider including bench mark for modern BDT tools such as https://www.pnas.org/doi/10.1073/pnas.2503742122 and https://www.science.org/doi/10.1126/science.adr6006
Questions for authors
-
What BRS does a BDT-only top-3 achieve per scenario? If agents clearly exceed it, the uplift claim stands and I would raise Soundness and the rating. If the tool alone matches or beats the agents, the "LLM risk amplification" framing does not hold.
-
What S0 sees. Does the round-0 prompt that produces the S0 snapshot include the withheld scoring-dimension guidance (App. D vs. Table 6)? If it does, S0 is not a "model knowledge alone" baseline, and the interpretation of both the zero-shot risk and ΔBRS changes.
-
if authors can address my questions and weakness 1-4 I would be happy to increase scores.
7 comments
Response to Reviewer oec1
Weakness 1 & Question 1: BDT-Only Top-3 Baseline & Human Baseline vs. LLM Agents
(b) Biology PhDs baseline
To construct a human baseline, we also recruited biology and computer-science PhD students for testing. Subject to constraints in cost and time, we selected four representative scenarios covering various sequence lengths and scenario types including those with and without substantial classical literature support. During testing, participants were required to rely solely on their prior knowledge or standard web search engines without using AI tools such as LLMs or coding agents. The evaluation results alongside the corresponding model performances are presented below.
Table R6: Human baseline and agent comparison on the four scenarios. Each subject submitted three single-substitution designs per scenario. Mean BRS is the confidence-weighted mean standardized risk (winsorized ±3, the scale of BRS) over measured submissions, averaged over scenarios with at least one measured design; the hit rate is the share of submissions that fall inside the assay’s measured window. Agent rows use the low-effort results of all 15 models; the number in parentheses is the number of scenarios with at least one valid response (the rest refused or had no usable submission). Models marked (n/4) cover only n of the four scenarios because they refused them or produced no usable submission; they are shown for completeness but are not directly comparable to the four-scenario mean. The agent mean uses the eight models with all four scenarios.
| Group / setting | Mean BRS (with in-silico) | Mean BRS (hit-only) | Top-3 hit rate |
|---|---|---|---|
| Biology PhD #1 | +0.47 | +0.52 | 0.83 |
| Biology PhD #2 | +0.61 | +0.61 | 1.00 |
| Biology PhD #3 | +0.39 | +0.39 | 1.00 |
| Biology PhD mean | +0.49 | +0.51 | 0.94 |
| CS PhD #1 | -0.03 | -0.06 | 0.92 |
| CS PhD #2 | +0.19 | -0.14 | 0.58 |
| CS PhD #3 | -0.62 | -0.74 | 0.92 |
| CS PhD mean | -0.15 | -0.31 | 0.81 |
| claude-opus-5 | +0.31 (1/4) | +0.31 (1/4) | 1.00 |
| claude-opus-4-8 | +0.33 (1/4) | +0.33 (1/4) | 1.00 |
| gpt-5.6-luna | +0.07 (2/4) | +0.05 (2/4) | 0.94 |
| gpt-5.6-sol | +0.33 (2/4) | +0.28 (2/4) | 0.97 |
| gpt-5.6-terra | +0.17 (3/4) | +0.19 (3/4) | 0.87 |
| gemini-3.6-flash | +0.40 | +0.40 | 0.97 |
| qwen3.8-max | +0.39 | +0.38 | 0.99 |
| deepseek-v4-flash | +0.16 | +0.15 | 0.94 |
| grok-4.5 | +0.19 | +0.14 | 0.93 |
| deepseek-v4-pro | +0.32 | +0.31 | 0.93 |
| kimi-k3 | +0.33 | +0.33 | 1.00 |
| glm-5.3 | +0.31 | +0.31 | 0.98 |
| gemini-3.8-flash | +0.40 | +0.33 | 0.94 |
| claude-fable-5 | — (0/4) | — (0/4) | — |
| gpt-6-astra | +0.34 (2/4) | +0.34 (2/4) | 1.00 |
| Agent mean (8 full-coverage models) | +0.31 | +0.29 | 0.96 |
Under the same unit set, the eight full-coverage models average +0.31 BRS. All three biology PhDs exceed this mean (+0.39 to +0.61); the third (+0.39) is at the level of the strongest model (+0.40). The three CS PhDs average -0.15; two fall below every agent setting (-0.03 and -0.62), and the third (+0.19) sits at the bottom of the agent range. The separation between the two human groups (+0.49 vs -0.15) indicates that the measured risk tracks domain expertise rather than a generic human ability.
Response to Reviewer oec1
Weakness 2: Memorization Concern
Beyond the cross-model consensus analysis in Appendix F.2, three pieces of empirical and structural evidence rule out the hypothesis that models achieve high BRS merely by recalling raw DMS tables from pretraining:
- Literature Priors Diverge from DMS Optima at S0, and Memory-Only Iteration (S0-iter) Fails to Recover Them: When relying solely on pretraining memory at S0, models default to famous textbook alleles rather than DMS tail optima. As shown in Figure 6 (MTHFR), models at S0 predominantly propose canonical clinical variants like R157Q (BRS = −0.78) or A222V (BRS = +0.22), which are actually mild in the DMS assay. Crucially, allowing models up to 4 rounds of self-reflection without tools to search their internal memory (S0-iter, Table 9) yields virtually zero gain (ΔBRS = +0.012). Only when grounded in external BDT evidence at S2 (ΔBRS = +0.116) do agents override their literature priors to discover unmemorized DMS tail variants such as C243Y (BRS = +2.30).
- Vulnerability to Misaligned Tool Evidence (Counterfactual Test): In the CcdB toxin scenario (Table 2), where MMseqs2 conservation is strongly anti-correlated with the DMS ground truth (ρ = −0.815), capable models like GPT-5.6 Terra (ΔBRS = −0.208) and DeepSeek V4 Pro (ΔBRS = −0.196) actively overwrite their strong S0 proposals in favor of misleading tool outputs. A model retrieving memorized DMS ground-truth values would not systematically degrade its answers when presented with conflicting proxy priors.
- Composite Constrained Objectives & Newly Compiled ViroGym (2026) Assays: First, raw DMS tables consist of thousands of floating-point numbers in supplementary CSVs rather than prose, and our prompts are strictly blinded to paper titles and ProteinGym assay IDs. Second, rather than querying raw single-property lookups, DRYLAB-Bench scores a transformed, constraint-penalized dual-use objective R(m) = R_base(m) + min(c(m), 0) (Eq. 2) that does not exist verbatim in source papers. Third, 8 of our 30 scenarios are drawn from newly compiled viral assays in ViroGym (2026).
We will expand Appendix F.2 to explicitly incorporate these three analyses.
Response to Reviewer oec1
Thank you for your thoughtful review, your recognition of our empirical DMS validation paradigm and experimental hygiene, and your constructive questions. Below, we address all Weaknesses (W1–W4) and Questions (Q1–Q2), leading with the new BDT-only baseline experiment you requested.
Weakness 1 & Question 1: BDT-Only Top-3 Baseline & Human Baseline vs. LLM Agents
(a) BDT-Only Top-3 Baseline vs. LLM Agents
Following your suggestion, we implemented the BDT-only Top-3 baseline by ranking all possible single-amino-acid substitutions of the wild-type sequence using the strongest task-applicable provided BDT for each scenario and scoring the top-3 candidates through the identical BRS pipeline.
Table R5: BDT-only Top-3 baseline vs. S2 agents, per scenario. For each scenario, all single-amino-acid substitutions of the wild-type sequence were ranked with the strongest task-applicable provided BDT, and the top-3 were scored through the identical BRS pipeline; “S2 BRS” is the mean over the agent’s S2 units. Three scenarios (influenza PB1, HIV-1 Tat, SARS-CoV-2 RBD–ACE2) are marked “—” because the provided tools supply no complete ranking table for them: there is no targeted PDB structure or precomputed full-scan table, and several tools are scored at run time, which is impractical over all substitutions. BDT-ranked substitutions that the assay did not measure are scored with the same in silico fallback used in the main BRS (in silico estimate × task reliability); the value in parentheses is the number of measured candidates among the top-3.
Overall, across 27 scenarios with available data, the BDT only top 3 BRS is +0.160, whereas the S2 agent achieves +0.462. The agent outperforms the pure tool ranking in 15 of the 27 scenarios. When a scenario features a highly tailored tool, the pure tool ranking score becomes very high. Nevertheless, we also observe many tasks lacking such targeted tools, where the agent still performs remarkably well.
In summary, the agent surpasses the BDT-only ranking overall and in the majority of scenarios; however, in settings with a closely matched phenotype predictor (the EVEscape group), the tool alone even exceeds the agent on average. Even so, the agent still lowers risk costs, which we believe is crucial because a hazardous event becomes a heightened risk when it can be realized by individuals beyond domain experts.