- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work introduces DARPinDB, a curated collection of 188 DARPin-target records across 37 targets, and DARPinBench, a library-ranking benchmark. 10 methods are evaluated on 10, 500-candidate libraries. The main findings are inconsistent early retrieval of annotated binders and examples where accurate complex prediction does not translate into favorable library rank. Two additional large-library and cross-target protocols are released without results.
Critical strengths
• The paper is generally easy to follow
• Establishing curated benchmarks is important for reliable evaluation and development of scoring models
• The evaluation compares several scoring families on shared candidates and reports per-target results, uncertainty, and computational costs.
Critical weaknesses
-
The negative labels limit the biological interpretation of the benchmark. This is my primary concern. Among 5,000 Mode 1 entries, only nine are experimentally confirmed nonbinders, all from BCL-W; 4,899 are synthetic candidates. Introducing one to three substitutions into a known binder does not establish loss of binding. Some variants could retain or improve binding, so ranking them highly may be useful experimentally while being penalized by the benchmark. Specific biological knowledge about this protein family as well as references to other fields where decoys are often created (pMHC benchmarks, DUD-E) would help to motivate the validity of these decoys and the methods used to generate them. A cheap construction audit should report pairwise Hamming distances (or other metrics) among each unit’s positives and each decoy’s distance to its second-nearest positive, where available. These would expose positive redundancy and overlapping mutation neighborhoods, although sequence proximity alone cannot establish whether a candidate binds.
-
The paper contributions are only partially demonstrated. The curated dataset is a useful contribution. However, two of the three proposed benchmark modes (modes 2 and 3) are released without experiments, leaving their usefulness for large-library screening and cross-target learning unexplored. At least inexpensive reference methods should be evaluated, and the provenance of the shared library should be documented. Modes 2 and 3 are not well described, not executed and provide no new insights as currently presented. The ten-method reference comparison is also useful, but the claimed failure diagnosis provides limited additional insight. Its central observation (that accurate pose recovery does not guarantee favorable library ranking) is consistent with previous work distinguishing structural accuracy from binding and affinity prediction in other protein settings. Demonstrating this behavior for DARPins adds application-specific evidence, but the manuscript should clarify what new understanding follows: does failure arise from pose generation, the selected scoring functions, or insufficient target-specific discrimination? The current examples largely document the distinction without explaining it.
Questions for authors
- Why must targets have an experimental co-crystal to enter the ranking evaluation? Could the authors include the remaining eligible targets in the screening benchmark while restricting DockQ analysis to the anchored subset? Given the additional binding annotations and the uncertainty arising from only ten evaluated units, this could substantially improve coverage and precision without weakening the structural controls.
- Do timing comparisons include MSA searches and prerequisite pose generation, and what computations were cached or reused? Generating identical MSAs for proteins differing by 1-3 mutations unnecessarily inflates computational costs for structure based models.
- How are the replacement amino acids chosen when generating mutation decoys? Please specify the allowed amino-acid alphabet and sampling distribution, and whether substitutions are constrained by position or physicochemical properties. Do the generated decoys follow the position-specific amino-acid preferences of the intended DARPin libraries? Introducing residues rarely observed at these positions could create shortcuts that models can take advantage of.
- Do the mutated residues lie at the DARPin-target interface in the predicted complexes? Reporting this fraction and its relationship to score changes would help distinguish incorrect interface placement from limited sensitivity to mutations within the interface.
- How do score changes and retrieval performance vary with mutation count () within each target? This would help characterize sensitivity to progressively larger sequence perturbations.
- Could the authors include random-ranking success rates alongside the Top-k results, accounting for each unit’s number of positives? This would help readers assess the improvement over chance.
2 comments
Dear Reviewer UWL3,
A gentle reminder that our revised PDF has been online since 13 November, with changes marked in blue. It includes the construction audit of the synthetic candidates, the position-profile baseline, and the mutation-stratified retrieval analysis you asked about.
The discussion period closes on 18 November (AoE), so time is getting short. If you have a moment, we would appreciate your feedback and, if the revision addresses your concerns, an updated assessment. If anything else should change, a note in the next day would let us address it before the deadline.
Thank you for your time.
Dear Reviewer UWL3,
Our revised PDF is now online, with changes marked in blue. Following your review, the construction audit of the synthetic candidates and the position-profile baseline are added; the random-ordering null appears in Figure 3b; Modes 2 and 3 have reference scores; a figure stratified by the number of substitutions (k) covers all ten methods; the timing paragraph in Section 4 states what each hour includes; and the Limitations section states the admission criterion for units.
We hope the revision addresses your concerns and look forward to your feedback. If anything else should be changed, we will do our best to address it before the deadline.
Thank you for the detailed and constructive review.