acceptodds
Under review as a conference paper at ICLR 2027

Humans vs. Autonomous Agents in De Novo Protein Binder Design: A Post-Competition Analysis of the muni x Adaptyv TREM2 Hackathon

Abstract

We present, to our knowledge, the first experimentally validated comparison of human-led and autonomous-agent de novo protein binder design. In a single-day hackathon, human teams and six agents submitted 81 and 60 designs under common constraints; 100 designs from nine human teams and six agents were tested. Of these, 89 expressed and 37 displayed binding to TREM2. Human and agent hit rates were 25/65 (38.5%) and 12/35 (34.3%), respectively (two-sided Fisher’s exact P=0.83), with no clear cohort advantage. The strongest binders had per-design mean values of 1.113 nM (human) and 3.636 nM (agent) by SPR. Experimental stability measurements were not available for the tested designs. Sequence-validated Protenix and Chai-1 predictions placed the strongest binder and all 36 mini-protein hits at TREM2’s hydrophobic tip, outside the principal crystallographic contact sites of scFv-2 and scFv-4; 46/49 expressed mini-protein non-binders were also predicted to contact this region. Retrospective Protenix ipSAE and ipTM distinguished binders from non-binders (AUROC 0.782 and 0.791) but correlated weakly with affinity among 36 fitted binders (Spearman rho=0.293 and 0.196), supporting binder triage but not reliable affinity ranking. A separate 15-round, human-directed agentic search produced ten candidates; nine yielded fitted BLI mean values below 10 nM and three below 1 nM. In this setting, the follow-up produced numerically stronger reported top-end affinities than the original hackathon, although the campaigns differed in duration, selection and assay. Anthropic’s subsequent Claude Science study selected TREM2 as a campaign target and benchmarked its autonomous binder-design results against the publicly released hackathon and agentic follow-up datasets. Together, these results establish the first experimentally validated benchmark of human and agentic protein design, showing similar observed hit rates under common submission constraints, while also highlighting the value of iterative multi-round agentic search.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
autonomous agents, protein design, AI agents, large language models, protein engineering, experimental validation, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LY6xOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This retrospective study compares human-led and autonomous workflows for TREM2 binder design. It reports 37 hits among 100 computationally selected submissions and evaluates predictive scores against experimental outcomes. A separate human-directed follow-up yielded nine candidates with fitted BLI affinities below 10 nM (Secs. 2-3). The main contribution is an experimentally measured benchmark.

Critical strengths

• Testing designs from nine human teams and six agents provides useful evidence of practical design outcomes. Sequence records, sensorgrams and sensitivity analyses support further investigation. Non-expression is distinguished from non-binding (Fig. 2 and Supplementary S1, S3, S5). • The analysis carefully separates binder discrimination from affinity ranking. Protenix's retrospective AUROCs approach 0.79, but its affinity correlations remain weak. The authors explicitly avoid interpreting similar observed hit rates as equivalence (Secs. 3.1-3.2).

Critical weaknesses
  1. Workflow provenance is insufficient to audit the central human-agent comparison. Prompts, chat histories, intervention logs and matched compute records are unavailable (Supplementary S1-S2). Differences in tools, workflows and possible agent use within human teams confound the autonomy comparison. One target and one event further limit generalization.
  2. Hit rates condition on Boltz-2 ipSAE selection of 100 from 141 submissions, with different selection fractions for human and agent entries (65/81 versus 35/60; Sec. 2.2). Design-level intervals omit team and lineage dependence (Sec. 2.4). The human-directed follow-up changes generator, length, selection and assay. It therefore cannot isolate the contribution of agentic iteration to the improved fitted affinities (Sec. 2.5; Supplementary S7). The measured resource has value, but incomplete provenance and confounded comparisons provide insufficient generalizable insight into autonomous binder design to support acceptance.
Questions for authors

Can workflow records support an auditable definition of autonomy and a reproducible workflow-level comparison?

How do team-aware, selection-sensitive analyses affect the comparison while treating untested submissions as unobserved?

1 comment

Thank you for the careful review.

Weakness #1 and Question #1: The cohorts differed in who chose the designs. The primary goal was to investigate the extent to which fully autonomous orchestration of protein design tools can produce high quality binders. As the submitted §2.1 explains, human teams could use AI tools. The six comparison agents received the same challenge prompt and start instruction. Each agent chose its scientific tools and selected its own ten submissions. The challenge brief set the global top-100 ipSAE rule in advance. Each human team and each agent had the same Muni compute allowance. Only agents have usage records, so the study cannot compare actual spending across cohorts (§§2.1–2.2).

S2 now includes the prompt and six workflow profiles. The supplementary ZIP contains a record of 172 scientific jobs with their order, outcomes and links to the submitted designs. For example, Claude used Boltz and ipSAE on all five PXDesign and five Foundry candidates it submitted. Grok used RF3 on two PXDesign submissions. Readers can trace these decisions and rerun the experimental analysis from the released data. Exact session replay would also require the original inference settings. Our comparison concerns the workflows used at this event on TREM2.

Weakness #2 and Question #2: The submitted §2.4 identified within-team dependence. S1 now reports an interval that accounts for it. Among tested designs, 25/65 human and 12/35 agent submissions bound, a human-minus-agent difference of 4.2 percentage points. Resampling whole teams within each cohort gave a 95% interval of −13.5-13.5 to +24.8+24.8 points (S1). Resampling keeps each team’s designs together. Related designs may still span teams.

We tested 65 of 81 human submissions and 35 of 60 agent submissions. Across all 141 submissions, the 41 unknown outcomes allow a human hit rate anywhere from 30.8 percentage points below the agent rate to 30.6 points above it. We therefore limit the hit-rate comparison to the tested panel, as in the original submission (§2.4).

Weakness #2: The follow-up began with the hackathon’s experimental results. The agent then completed 15 computational rounds before BLI testing. Nine of its ten candidates yielded fitted mean affinities below 10 nM. The original abstract explicitly identified the differences in duration, selection and assay, and §3.4 distinguished follow-up BLI from hackathon SPR. We report what the agent selected and what BLI measured. A claim that agentic iteration caused the lower fitted affinities would require a matched search baseline and comparable assays. The paper makes no such claim.

Reviewer_CZ82OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the performance of human-led teams and autonomous AI agents in designing de novo protein binders targeting TREM2, a receptor associated with neuroinflammation and Alzheimer's disease. The study consists of three main components. First, the authors analyze a controlled-submission protein-design competition. Nine human teams comprising 40 participants and six autonomous AI agents participated in a single-day hackathon. Participants were asked to design binders targeting the IgSF domain of TREM2 under common sequence and submission constraints. Human teams submitted 81 designs, while agents submitted 60. The 100 highest-scoring designs, selected using Boltz-2-derived ipSAE scores, underwent experimental validation. Among the tested designs: Metric Human teams Autonomous agents Designs tested 65 35 Successfully expressed 60 (92.3%) 29 (82.9%) Binding hits 2 5 (38.5%) 12 (34.3%) Strongest measured binder 1.11 nM 3.64 nM The difference in binding-hit rates is not statistically significant (Fisher's exact test, p=0.828p=0.828). The authors appropriately note that this does not establish equivalence between the two cohorts. Second, the authors evaluate computational confidence metrics and predicted binding interfaces. They compare Boltz-2, Protenix, and Chai-1 predictions against experimental outcomes. Retrospective Protenix ipSAE and ipTM achieve AUROCs of 0.782 and 0.791 for distinguishing binders from non-binders, but correlate weakly with measured binding affinity. Structural predictions suggest that the strongest binders interact with a hydrophobic region of TREM2 distinct from the principal crystallographic contact sites of two previously characterized antibodies. However, similar predicted contacts also occur in most non-binding designs, limiting their discriminative value. Third, the paper reports a post-hackathon agentic computational search campaign. Using NVIDIA's Proteina-Complexa framework, a human-directed agent conducts 15 rounds of computational design and evaluation, varying design parameters and selecting promising candidates for experimental testing. Of ten selected designs, nine yield fitted BLI binding affinities below 10 nM, including three with mean affinities of 0.232, 0.583, and 0.619 nM. The authors conclude that autonomous agents can produce experimentally validated protein binders with observed success rates comparable to those of human-led teams, while iterative agentic search offers a promising direction for extending computational protein design.

Critical strengths
  1. Experimentally grounded evaluation of autonomous scientific agents. The most significant strength of the paper is its use of real experimental measurements to evaluate autonomous agents. Many existing agent benchmarks focus on code execution, simulated environments, or tasks evaluated by other models. In contrast, this study assesses whether agent-generated protein sequences can be expressed and bind a biological target. This provides a more meaningful evaluation of scientific usefulness than purely computational metrics.
  2. Direct comparison between human-led teams and autonomous agents. The hackathon provides an interesting setting for comparing human and autonomous workflows under common submission constraints. Both cohorts target the same protein domain, follow the same sequence restrictions, and participate in the same experimental selection process. Although the comparison has important confounding factors, it represents a useful initial attempt to benchmark autonomous scientific agents against human-led teams using experimental outcomes.
  3. Valuable empirical finding that autonomous agents can produce functional binders. The results demonstrate that autonomous agents can complete a substantial protein-design workflow and generate experimentally validated binding proteins. The agent cohort achieves 12 binding hits among 35 tested designs, with the strongest measured binder reaching a reported affinity of 3.64 nM. This is a meaningful demonstration of practical capability, even though the study does not establish parity or superiority over human teams.
  4. Careful distinction between binder classification and affinity prediction. The analysis of computational confidence metrics is particularly informative. Figure 3 (page 6) shows that retrospective Protenix scores can distinguish binders from non-binders reasonably well, with AUROCs around 0.78–0.79, while exhibiting weak correlations with measured affinity among successful binders. This distinction has important implications for computational protein design: a score that enriches a candidate pool for binding hits is not necessarily suitable for optimizing binding strength. The finding is relevant beyond the specific human-versus-agent comparison.
  5. Promising results from multi-round computational search. The post-hackathon campaign provides evidence that an agent-assisted workflow can sustain multiple rounds of computational design, evaluation, and parameter adjustment. The reported subnanomolar binders are promising, and Figure 5 (page 8) provides a useful overview of the computational workflow and experimental follow-up. This component points toward a potentially important application of AI agents: coordinating repeated computational experiments rather than generating a single set of designs.
  6. Generally careful statistical reporting and acknowledgment of limitations. The paper reports confidence intervals, statistical tests, multiple-comparison adjustments, and distinctions between experimental outcomes and computational predictions. The authors explicitly acknowledge that the observed hit rates do not establish equivalence, that structural predictions do not validate the binding epitope, and that different experimental assays prevent direct comparison of affinity values across campaigns. This improves the credibility of the empirical analysis.
Critical weaknesses
  1. The human-versus-agent comparison is confounded by differences in design tools. This is my primary concern. Although the competition imposes common submission constraints, it does not control the computational tools used by the two cohorts. According to Section 3.1, all autonomous agents use PXDesign, whereas human teams employ a broader range of generative, hallucination, optimization, and diversification workflows. Consequently, the observed outcomes reflect a combination of the underlying design software, computational resources, candidate selection, and human or agent decision-making. It is therefore difficult to attribute the results specifically to autonomous agents. A stronger comparison would evaluate human-led and agent-led workflows using the same underlying design tools and comparable computational budgets.
  2. The comparison does not establish equivalence between human and agent performance. The paper reports human and agent hit rates of 38.5% and 34.3%, with a nonsignificant difference. However, failure to reject a difference does not imply that the two approaches perform equivalently. The reported 95% bootstrap confidence interval for the hit-rate difference ranges from −15.2 to 23.5 percentage points, allowing for potentially substantial advantages in either direction. Furthermore, the sample sizes are relatively small, and designs from the same team or generation workflow may be correlated. The evidence therefore supports the narrower conclusion that both cohorts successfully generated binders and that no statistically significant difference was detected in this experiment.
  3. Experimental selection introduces substantial selection bias. Only 100 of the 141 submitted designs were selected for experimental testing, using a computational confidence metric. The tested proportions differ between the two cohorts: 65/81 human submissions and 35/60 agent submissions. As a result, the reported hit rates are conditional on a nonrandom selection procedure. This is particularly important because human submissions have significantly higher computational confidence scores than agent submissions. The selection process may therefore affect the composition of the tested cohorts in different ways. Without testing all submissions, or applying a sampling strategy designed for unbiased cohort comparison, the study cannot reliably estimate the overall experimental success rates of human-led and autonomous design workflows.
  4. The paper provides limited insight into the autonomous agents' reasoning and decision-making. Although the central theme concerns autonomous agents, the analysis focuses primarily on final protein sequences and experimental outcomes. The paper provides limited information about how individual agents planned their workflows, selected tools, evaluated candidate designs, recovered from failures, or allocated computational resources. As the authors acknowledge, the final designs do not reveal whether agents made and corrected errors during the design process. This is a significant limitation for an ICLR submission because the paper does not clearly identify which agentic capabilities contributed to successful protein design. A more detailed analysis of agent trajectories, decisions, and failure modes would substantially strengthen the contribution.
  5. The post-hackathon campaign lacks an appropriate optimization baseline. The 15-round agentic search campaign produces promising binders, but the paper does not compare it against conventional computational search methods under matched conditions. For example, it is unclear whether similar or better results could be obtained using random search, Bayesian optimization, evolutionary optimization, or a manually configured PXDesign workflow with an equivalent computational budget. The improvement in top-end reported affinity may result from increased search effort, different design settings, or selection procedures rather than agentic reasoning. Without an appropriate baseline, the study demonstrates that the workflow can find strong binders, but not that autonomous iterative search provides a measurable advantage over simpler alternatives.
  6. The post-hackathon campaign is not a closed-loop experimental optimization study. The distinction between computational iteration and experimental feedback is important. According to Section 2.5 and Figure 5, the 15 search rounds involve computational generation, scoring, and parameter adjustment. The selected designs are then tested experimentally. The manuscript does not report experimental binding measurements being fed back into the agent during those 15 rounds. Therefore, the experiment demonstrates multi-round in silico optimization followed by experimental validation, rather than a fully autonomous design-build-test-learn loop with iterative wet-lab feedback. This distinction should be emphasized when discussing the implications for autonomous scientific discovery.
  7. Affinity results from the two campaigns are not directly comparable. The hackathon designs are primarily characterized using SPR, whereas the post-hackathon designs are measured using BLI. The two campaigns also differ in duration, candidate-generation procedures, and selection criteria. Although the post-hackathon campaign produces numerically stronger reported affinities, these differences prevent a controlled conclusion that the iterative agentic approach improved binding affinity relative to the hackathon. A more convincing comparison would evaluate representative designs from both campaigns using the same assay and measurement protocol.
Questions for authors

Q1. How can the authors separate the contribution of autonomous agents from that of the underlying protein-design tools? All autonomous agents use PXDesign, whereas human teams use several different design methods. Could the authors clarify whether any human teams used PXDesign under comparable settings? Would it be possible to conduct a controlled comparison in which human and autonomous workflows use the same design software and computational budget? Q2. How should the observed hit-rate comparison be interpreted given the experimental selection procedure? Only 100 of 141 submitted designs were tested, and the tested proportions differ between human and agent cohorts. Could the authors provide an analysis of how the Boltz-2-based selection procedure may bias the observed cohort hit rates? Is there a way to estimate the sensitivity of the conclusions to the untested submissions? Q3. What specific capabilities of autonomous agents contributed to successful binder design? The paper establishes that agents can generate successful designs, but provides limited analysis of their internal decision-making. Could the authors provide representative agent trajectories illustrating task planning, tool selection, candidate evaluation, error recovery, and design refinement? Which of these capabilities were necessary for successful outcomes, and which were already handled by the underlying design software?

1 comment

Thank you for recognizing the value of testing agent-designed proteins in the lab. Your question about PXDesign prompted us to correct the tool inventory and report the within-tool counts. They clarify both the experimental result and the limits of attributing it to the agents.

Weakness #1 and Question #1: The submitted §4.3 correctly identified tool choice as a confounder, but its description of agent tool use was incomplete. Agents submitted designs from four workflows: PXDesign, Foundry, BoltzGen and RFpeptides. All 12 agent hits came from PXDesign. Among tested PXDesign-family submissions, 8/11 human and 12/27 agent designs bound. For the exact PXDesign label, the counts were 1/2 and 12/27. Each human team and each agent received the same Muni allowance and could use external tools. Only agent usage records were available. S1 now reports the family counts. Isolating the effect of agent decision-making would require a comparison with matched settings, selection and compute use.

Weakness #2: We addressed dependence within teams by resampling whole teams within each cohort. This gave a 95% interval of −13.5-13.5 to +24.8+24.8 points around the observed 4.2-point difference. With nine human teams and six agents represented in testing, and possible shared lineages across teams, equivalence is unsupported. Both cohorts did produce single-digit-nanomolar binders. The revised abstract states both conclusions (§2.4, S1).

Weakness #3 and Question #2: The submitted §2.4 limited the comparison to selected designs. A common Boltz-2 ipSAE shortlist advanced 100 of 141 submissions. In that panel, 25/65 human designs (38.5%) and 12/35 agent designs (34.3%) bound. The 16 untested human designs and 25 untested agent designs have unknown outcomes. Assigning those outcomes to favor each cohort in turn gives human-minus-agent differences from −30.8-30.8 to +30.6+30.6 percentage points across all submissions. The different selection fractions therefore leave even the direction of the full-cohort difference open.

Weakness #4 and Question #3: The workflow records show which decisions the agents made. Claude tried six generation tools, including a failed BindCraft run. It used Boltz to predict the candidate–TREM2 complex and ipSAE to score the interface for each of its ten submissions. Grok used RF3 on two PXDesign candidates it later submitted. Claude and GPT began evaluating while generation was running. Qwen and Gemini began separate evaluations after generation finished. The agents chose the tools, timing and submission panels. The tools performed generation, prediction and scoring. S2 and the accompanying workflow record document all six trajectories, including failed jobs and submission links. Establishing which decisions were necessary for binding would require varying those decisions experimentally.

Weakness #5: Whether the agent’s search policy outperforms random, Bayesian, evolutionary or manually configured search at equal compute remains an open question. However, baselines such as Bayesian optimization are not necessarily well-suited to the question being asked here. The agents and human teams operated in an open-ended design space. Defining the search space for baseline methods would require introducing assumptions and biases that would affect the outcome. What the experiment establishes is that the agent selected ten designs, nine of which yielded fitted mean BLI affinities below 10 nM, including three below 1 nM.

Weakness #6: The tested hackathon designs and their outcomes were inputs to the follow-up. As the submitted §2.5 described, the 15 rounds were computational. Candidates were generated and scored during the search, and ten selected designs then underwent BLI testing. Figure 5 now makes clear that its return arrow illustrates a prospective experimental cycle. No wet-lab measurements were fed back during the 15 rounds.

Weakness #7: The original §3.4 identified the leading hackathon values as SPR measurements, and the abstract flagged differences in duration, selection and assay. We retain the numerical comparison on those terms. Figure 5 labels the two assays separately, and S7 gives the fit qualifications. Establishing a gain in affinity attributable to the search strategy would require a matched comparison.

Reviewer_ropvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes the muni x Adaptyv TREM2 binder design hackathon (Feb 2026). Nine human teams and six autonomous LLM agents submitted 81 and 60 designs against the TREM2 IgSF domain under common rules. The top 100 by Boltz-2 ipSAE were expressed cell-free and tested, mostly by SPR: 89 expressed and 37 bound. Hit rates were 38.5% (human) and 34.3% (agent), Fisher p=0.83, with best binders at 1.11 nM and 3.64 nM.

Three secondary analyses follow. Retrospective Protenix ipSAE/ipTM separate binders from non-binders (AUROC 0.78/0.79) but correlate weakly with affinity among the 36 fitted binders (rho 0.29/0.20). All 36 mini-protein hits are predicted to contact the TREM2 hydrophobic tip, but so are 46/49 expressed non-binders, so the prediction does not discriminate. A separate 15-round agentic search produced ten BLI-tested designs, three reported below 1 nM.

Critical strengths

S1. The dataset. Wet-lab-labeled binder design data is expensive and rare, and this one is released. Two things are done better than in most competition datasets: non-expression and expressed-non-binding are annotated separately, and the pre-experimental scores are kept apart from the retrospective Protenix/Chai-1 predictions (Sec 2.2). The data is useful regardless of whether the paper's main claim holds.

S2. The confidence-score analysis in Sec 3.2. Treating binder discrimination and affinity ranking as separate endpoints is the right call, and the gap is large: AUROC 0.78 but rho 0.29. The practical reading - ipSAE can enrich a panel but cannot order candidates within it - is backed by experimental labels rather than in-silico proxies. The observation that the same score differs across predictors (Boltz-2 ipSAE 0.652 vs Protenix 0.782) is also worth having. I credit the authors for reporting the 46/49 non-binder result in Sec 3.3 instead of showing only the convergence of the hits.

Critical weaknesses

W1. The human-vs-agent comparison is not rigorous enough to support the claim in the title.

  • Tool access is never characterized. The best human design used Mosaic, all agent entries in Fig 1 used PXDesign. Whether both cohorts could reach the same tools is not stated anywhere. Same rules is not the same as same action space.
  • Tool choice is fully confounded with cohort (Sec 4.3). This is closer to "mixed toolchains vs PXDesign" than to "humans vs agents". The analysis that would separate them, human PXDesign vs agent PXDesign, is missing.
  • Selection truncated the two pools differently. Top-100 was a single global ipSAE ranking and human scores were higher (p=0.00085), so 80.2% of human submissions were tested against 58.3% of agent submissions. Sec 3.1 compares these as if they were cohort samples.
  • The tests assume 100 independent designs, but they sit in 15 units with up to ten sequences each, usually from one pipeline. Effective n is closer to 9 vs 6. Sec 2.4 admits this and the analysis is unchanged.

The title and abstract say "similar hit rates", which reads a failure to reject as equivalence. The paper's own interval is -15.2 to +23.5 pp and no equivalence test is run. Sec 3.1 is careful about this; the abstract is not. The last two points can be fixed by reanalysis. The first two cannot, so the claim should be withdrawn rather than repaired.

W2. Under-specified as an empirical contribution.

  • The hit criterion is never defined in the main text. No threshold, no K_D ceiling, no fit-quality filter. So 38.5% and 34.3% cannot be reproduced or compared with other campaigns.
  • No negative or specificity controls. No scrambled sequence, no irrelevant target. The false positive rate of the hit label is unknown, which matters when hit rate is the primary endpoint.
  • Sec 2.1 gives the agents two sentences. No scaffold, tool list, compute or time budget, prompt, or trajectory, and Sec 4.4 says the records are incomplete. For a paper about agents the trajectories are the most valuable artifact. Without them the data supports claims about outcomes but not about agent behaviour, and the outcome claims are limited by W1.
  • The sub-nanomolar values in Sec 3.4 are probably not identifiable. The BLI series is 0, 31.6, 100, 316.2, 1000 nM, so the lowest non-zero point is ~140x above the reported 0.232 nM. Under a 1:1 model every concentration would be above 99% occupancy, and dissociation in Fig 5c is nearly flat. These should be upper bounds, not 0.232 +/- 0.113 nM. This affects the section title and Fig 5b, which also puts BLI and SPR on one axis.
Questions for authors
  1. Did both cohorts have access to the same tools? If not, what are the human hit rate and top affinity after removing designs made with tools the agents could not use?
  2. What defines a binding hit? If the criterion came from the assay platform, please state it as reported.
  3. Hit rate for human PXDesign designs vs agent PXDesign designs?
  4. Please clarify provenance. Sec 2.2 refers to a publicly released scoring pipeline, but the AI Usage Statement says measurements came from your own laboratory records. Which data was generated for this work?
  5. Did both cohorts finish within the single day? Which designs were SPR and which BLI, and how was the canonical K_D picked when both existed?
  6. For Sec 3.4, please give k_a, k_d with fit uncertainties, residuals, and steady-state vs kinetic agreement. Is K_D identifiable at 31.6 nM minimum concentration?
  7. The Claude Science study is cited but its TREM2 numbers are not given. They speak to whether this measures agent capability or measures the one-day setting. Please report and discuss them.
1 comment

Thank you for the careful review. We concluded in §3.1 of the original submission that the observed hit rates do not establish equivalent performance. The revised abstract now also states that conclusion alongside the rates.

Weakness #1 and Questions #1 and #3: Both cohorts could use external tools, and each human team and each agent received the same Muni compute allowance. MRAZS used private code built around the public Mosaic package, as the rules allowed. We therefore retain its 1.113 nM binder. The agents submitted designs from PXDesign, Foundry, BoltzGen and RFpeptides. All 12 agent hits came from PXDesign. Among tested PXDesign-family designs, 8/11 human and 12/27 agent designs bound. Using the exact PXDesign label gives 1/2 and 12/27, respectively. We now report the family comparison in S1.4. Teams chose their own settings and candidates, so these counts cannot isolate the contribution of the human or agent (§2.1).

Weakness #1: Each team chose its submissions. A single ipSAE shortlist determined which 100 of 141 designs went to the lab: 65/81 human and 35/60 agent designs. The observed hit rates, 25/65 and 12/35, apply to that tested panel. The other 41 outcomes are unknown. We addressed within-team dependence by resampling whole teams within each cohort. The human-minus-agent difference was 4.2 percentage points, with a 95% interval of −13.5-13.5 to +24.8+24.8 points. The interval includes advantages for either cohort. Resampling accounts for clustering within teams. Related designs may still span teams (§2.4, S1).

Weakness #2 and Question #2: Both hit rates follow from the released labels. We counted Adaptyv’s “binder” and “weak” labels among all tested designs, including non-expressers, with no additional affinity cutoff or fit-quality filter (§2.3). For curves without a reliable fit, Adaptyv’s published criterion is an association shift “≥300%\geq 300\% over the negative control.” We report that as platform guidance. S3 documents zero-concentration traces for all 22 hackathon BLI replicates. The submitted §4.4 limited our claim to binding under the assay conditions. The dataset contains no scrambled-sequence or unrelated-target measurements that would establish specificity or calibrate the hit-label false-positive rate. SPR methods, BLI methods

Weakness #2 and Question #5: Each team and each agent completed its design work within one day. S2 now gives the shared prompt, start instruction and six workflow profiles. Each agent chose its tools and ten submissions. Claude tried six generation tools, including a failed BindCraft run, and submitted five PXDesign and five Foundry designs. Across the six agents, we estimate USD 114.18 for 172 scientific jobs and USD 18.53 for LLM use, totaling USD 132.71 (Tables S2.2–S2.3). Usage records cover agents only.

Questions #4 and #5: Adaptyv Bio measured expression and binding for both campaigns. We scored submissions with Adaptyv’s public pipeline and ran the retrospective predictions separately. Data S5 identifies the assay for every hackathon replicate. We averaged the included fitted affinities for each design. The means for designs 22, 63, 87 and 134 include both SPR and BLI fits (Table S3.3).

Weakness #2 and Question #6: The lowest tested concentration does not by itself turn a kinetic estimate into an upper bound. Zhao et al. reported a fitted 1.78 nM from a 31.25–250 nM BLI series. For AR03, the limiting evidence is its nearly flat dissociation trace, already flagged in S7. Data S4 gives fitted association rates of 66,377 and 32,039 M−1 s−1\mathrm{M}^{-1}\,\mathrm{s}^{-1}, with both off-rates near 10−5 s−110^{-5}\,\mathrm{s}^{-1}. The replicate SD measures variation between fits. Parameter uncertainties, full residuals and a steady-state comparison are unavailable. We retain 0.232±0.1130.232 \pm 0.113 nM as the mean ±\pm SD of the two fits. The traces cannot support a precise off-rate or a numerical upper bound. Figure 5 identifies the assays, and Table S7.3 gives the available fit errors. Zhao et al., Fig. 2d

Question #7: We added the TREM2 results to §4.3. Claude Science reported 72/90 binders across three 24–48-hour campaigns using combined Adaptyv and Twist results, or 65/90 using Adaptyv alone. Their retest of muni-AR-03 as a positive control showed binding below their 100 pM assay floor. The retest independently supports strong AR03 binding. It cannot quantify the precision of the original off-rate. Models, resources, selection and hit criteria also differed, so this comparison cannot isolate the effect of duration. Claude Science report, p. 14

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.