- Rating
- 1
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This retrospective study compares human-led and autonomous workflows for TREM2 binder design. It reports 37 hits among 100 computationally selected submissions and evaluates predictive scores against experimental outcomes. A separate human-directed follow-up yielded nine candidates with fitted BLI affinities below 10 nM (Secs. 2-3). The main contribution is an experimentally measured benchmark.
Critical strengths
• Testing designs from nine human teams and six agents provides useful evidence of practical design outcomes. Sequence records, sensorgrams and sensitivity analyses support further investigation. Non-expression is distinguished from non-binding (Fig. 2 and Supplementary S1, S3, S5). • The analysis carefully separates binder discrimination from affinity ranking. Protenix's retrospective AUROCs approach 0.79, but its affinity correlations remain weak. The authors explicitly avoid interpreting similar observed hit rates as equivalence (Secs. 3.1-3.2).
Critical weaknesses
- Workflow provenance is insufficient to audit the central human-agent comparison. Prompts, chat histories, intervention logs and matched compute records are unavailable (Supplementary S1-S2). Differences in tools, workflows and possible agent use within human teams confound the autonomy comparison. One target and one event further limit generalization.
- Hit rates condition on Boltz-2 ipSAE selection of 100 from 141 submissions, with different selection fractions for human and agent entries (65/81 versus 35/60; Sec. 2.2). Design-level intervals omit team and lineage dependence (Sec. 2.4). The human-directed follow-up changes generator, length, selection and assay. It therefore cannot isolate the contribution of agentic iteration to the improved fitted affinities (Sec. 2.5; Supplementary S7). The measured resource has value, but incomplete provenance and confounded comparisons provide insufficient generalizable insight into autonomous binder design to support acceptance.
Questions for authors
Can workflow records support an auditable definition of autonomy and a reproducible workflow-level comparison?
How do team-aware, selection-sensitive analyses affect the comparison while treating untested submissions as unobserved?
Thank you for the careful review.
Weakness #1 and Question #1: The cohorts differed in who chose the designs. The primary goal was to investigate the extent to which fully autonomous orchestration of protein design tools can produce high quality binders. As the submitted §2.1 explains, human teams could use AI tools. The six comparison agents received the same challenge prompt and start instruction. Each agent chose its scientific tools and selected its own ten submissions. The challenge brief set the global top-100 ipSAE rule in advance. Each human team and each agent had the same Muni compute allowance. Only agents have usage records, so the study cannot compare actual spending across cohorts (§§2.1–2.2).
S2 now includes the prompt and six workflow profiles. The supplementary ZIP contains a record of 172 scientific jobs with their order, outcomes and links to the submitted designs. For example, Claude used Boltz and ipSAE on all five PXDesign and five Foundry candidates it submitted. Grok used RF3 on two PXDesign submissions. Readers can trace these decisions and rerun the experimental analysis from the released data. Exact session replay would also require the original inference settings. Our comparison concerns the workflows used at this event on TREM2.
Weakness #2 and Question #2: The submitted §2.4 identified within-team dependence. S1 now reports an interval that accounts for it. Among tested designs, 25/65 human and 12/35 agent submissions bound, a human-minus-agent difference of 4.2 percentage points. Resampling whole teams within each cohort gave a 95% interval of −13.5 to +24.8 points (S1). Resampling keeps each team’s designs together. Related designs may still span teams.
We tested 65 of 81 human submissions and 35 of 60 agent submissions. Across all 141 submissions, the 41 unknown outcomes allow a human hit rate anywhere from 30.8 percentage points below the agent rate to 30.6 points above it. We therefore limit the hit-rate comparison to the tested panel, as in the original submission (§2.4).
Weakness #2: The follow-up began with the hackathon’s experimental results. The agent then completed 15 computational rounds before BLI testing. Nine of its ten candidates yielded fitted mean affinities below 10 nM. The original abstract explicitly identified the differences in duration, selection and assay, and §3.4 distinguished follow-up BLI from hackathon SPR. We report what the agent selected and what BLI measured. A claim that agentic iteration caused the lower fitted affinities would require a matched search baseline and comparable assays. The paper makes no such claim.