- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates antimicrobial peptide (AMP) modification as a parent-relative decision: whether to modify a peptide, where to edit it, and which chemical structure to introduce. AMP-Modify combines an LLM-assisted search controller, a registry of 20 chemical edit types, a deterministic molecular compiler, and an AMP-specific evaluator that jointly represents peptide sequence, molecular structure, and edit events. The evaluator is trained using a curated corpus of 8,145 parent–modified relations, of which 980 high-confidence pairs provide direct pair-effect supervision. Reported evaluations include held-out pair ranking, ten literature cases, and seven retrospective search tasks. The framework improves construction efficiency and evaluator-qualified search yield over several controls; however, the experimental MIC support for final search recommendations remains indirect.
Critical strengths
-
Well-motivated parent-relative formulation. Predicting whether an explicit edit benefits its parent is more decision-relevant than merely predicting whether the modified peptide remains antimicrobial. Treating retention of the parent as a possible outcome also makes the problem formulation practically meaningful.
-
Executable, auditable chemical design. Separating LLM proposals from rule-governed construction and downstream evaluation reduces structurally invalid and duplicate candidates. Connected-component data splitting and the unusually detailed appendix provide a useful starting point for rigorous verification of this modular workflow.
Critical weaknesses
-
Uncertain validity of the supervision and comparative evidence. Organism-group MIC aggregation does not, by itself, guarantee assay-comparable parent–variant effects. The C7 literature classification appears inconsistent with the manuscript’s molar two-fold threshold, and the stated Termini baseline architecture differs from its source. These issues affect the reliability of both targets and reported gains.
-
The central closed-loop claim lacks an independent outcome test. The evaluator both guides search and determines which generated candidates count as improvements. Consequently, 57.3% evaluator-qualified yield does not establish a corresponding gain in measured antimicrobial potency. The actual final, post-filtered top-K recommendations—and whether retaining the parent is the right choice—are not directly validated against measured outcomes.
Questions for authors
Q1. Pair-label provenance and the C7 discrepancy. How are parent and modified-peptide MIC values matched before organism-group medians are formed—at the level of strain, study, and assay conditions—and how do results change on strictly matched pairs? In C7, the reported IDR1018 / PEG6-IDR1018 values (16 → 32 μg/mL; molecular weights 1535.8 / 1872.3 g/mol) convert to approximately 10.42 → 17.09 μM, a 1.64-fold increase rather than the two-fold molar increase required for “worsen” (Tables 2 and 8; Appendix C.1). What exact records justify that label? If these are the evaluated endpoints, please correct C7 and the nine-case summary. Please also quantify the effect of censored MIC substitutions on labels near the threshold.
Q2. Fidelity and fairness of baseline comparisons. Was Termini implemented as its original Chemprop graph model with Morgan-count features, or as the Morgan- count logistic classifier described in Appendix D.2? Please provide the exact model revisions, configuration, and output-to-pair-score mapping. If the latter is a simplified substitute, distinguish it from the original Termini architecture. Please also clarify whether organism-group information is supplied comparably to pairwise controls and what score is used in the “without pair loss” ablation, for which a mostly unsupervised pair head would not be a task-matched control.
Q3. Independent validation of the feedback loop. Can the claimed benefit of evaluator feedback be demonstrated on experimentally measured effects that are independent of the evaluator and unavailable through runtime retrieval? The 57.3% versus 48.4% strict-improve yields in Table 3A are defined by the same frozen evaluator used to steer the search, so they measure agreement with that evaluator rather than measured MIC improvement. A blinded retrospective benchmark on independently measured parent–variant series—with unmeasured proposed candidates explicitly separated— would directly assess whether feedback improves true enrichment.
Q4. Quality of deployed recommendations and abstention. What are the actual post-gate top-K outputs, and how often do experimentally beneficial edits enter those final lists? Appendix F.8 places exact C5, C8, and C10 literature edits at ranks 55, 55, and 84 in the full candidate pool, not in the improve-qualified shortlist; these ranks alone cannot establish whether the edits are recommended. Please report qualified ranks and compare the current MIC-activity-probability ordering with parent-relative alternatives. Please also evaluate cases where the appropriate decision is to retain the parent, including false-positive modification recommendations.
Q5. Exactness of chemical structures and claimed coverage. How sensitive are the literature-case conclusions to missing or approximate chemical representations? Appendix E.2 states that C4 and C6 lack complete molecular graphs and that C7, C8, and C10 use proxies; in particular, the C8 head-to-tail-cycle proxy differs from the reported side-chain cyclization. Can the authors use exact, stereochemically specified structures wherever available, rerun the relevant scores, and provide matched SMILES/event records? Independently checked examples should also demonstrate that the compiler constructs the intended edit, beyond passing graph-syntax checks.
Q6. Added value and repeatability of the LLM search policy. How much improvement is attributable specifically to LLM-guided strategy selection rather than shared registryconstraints, compilation, deduplication, allocation, and feedback? Please compare against a non-LLM policy with the same registry, compiler, prior evidence, candidate budget, and stopping rules, and distinguish usable candidates from actual measured improvements. Because the present search settings are run once per case with unspecified provider decoding defaults (Appendix F), repeated runs and per-case variability are important for interpreting the reported differences.
12 comments
Response to Reviewer nooZ, Q2(1/2)
Q2. Question: Fidelity and fairness of baseline comparisons. Was Termini implemented as its original Chemprop graph model with Morgan-count features, or as the Morgan-count logistic classifier described in Appendix D.2? Please provide the exact model revisions, configuration, and output-to-pair-score mapping. If the latter is a simplified substitute, distinguish it from the original Termini architecture. Please also clarify whether organism-group information is supplied comparably to pairwise controls and what score is used in the ``without pair loss'' ablation, for which a mostly unsupervised pair head would not be a task-matched control.
Answer:
We thank the reviewer for raising this point. We used a simplified adaptation, not the original Chemprop architecture, for the Termini result in the original Table 1A. Specifically, it uses 2048-dimensional Morgan-count fingerprints (radius 2), organism-group one-hot features, StandardScaler, and class-balanced logistic regression (), retrained on our connected-component folds. To test whether this simplification accounts for the observed performance gap, we additionally retrained a Chemprop D-MPNN with Morgan-count features following Termini's published architecture. This implementation uses Chemprop 1.5.2, seven organism-group classification outputs, and 10 training epochs under the same five-fold protocol. We explicitly distinguish it from the original species-specific implementation (Chemprop 1.7.1) rather than claiming an identical reproduction. The D-MPNN obtains a five-seed family-balanced Spearman of on 951 scorable Tier-A pairs, compared with for the logistic adaptation (Table 1). Revised Table 1A reports this D-MPNN as Termini. The logistic substitute is appendix-only. The 29 gaps are pairs that a SMILES graph model cannot ingest, typically PEG, Fmoc, or metallocene edits, not pairs dropped from the benchmark. These additional results indicate that the main performance difference is not simply explained by replacing the graph architecture with logistic regression.
We provide organism-group information to the applicable activity-based controls. Termini-LogReg uses one-hot features, Termini-Chemprop uses group-specific output tasks, and AMPGAN/AMPGAN-e use organism-group indices. Our evaluator incorporates the group identity directly into its supervised pair head. For both Termini implementations, the relative score is , where denotes the organism group. This difference is aligned with (modified minus parent). AMPGAN similarly derives pair scores from differences in absolute activity predictions. These methods do not train a dedicated pair-effect head. DeepDelta directly predicts molecular property differences but does not receive organism-group information in our implementation. The revised appendix records these distinctions.
We agree that the original ``w/o pair loss'' evaluation was not a task-matched absolute-MIC control. The reported family-balanced Spearman of (Pair Spearman ) was computed from a pair head whose supervision was disabled. We therefore additionally score the same ablated model using the parent--modified difference from its trained absolute MIC head, obtaining an MIC- Spearman of , compared with a pair-head Pair Spearman of for the full model (Table 2). Revised Table 1B reports both rows. The MIC- score is the appropriate absolute-prediction control. The unsupervised pair-head result is retained only as a diagnostic, rather than being used to argue that absolute activity prediction cannot recover relative effects. Checkpoint selection also switches from validation pair performance to validation MIC F1 when pair supervision is removed. These corrections provide a more faithful assessment of the contribution of explicit pair supervision.
Response to Reviewer nooZ, Q2(2/2)
Table 1. Comparison of the two Termini implementations (five-seed mean standard deviation). Revised Table 1A reports the D-MPNN. The 29 Chemprop gaps are pairs that a SMILES graph model cannot ingest, not pairs dropped from the benchmark.
| Implementation | Coverage | Family-bal. Sp. | Pair Sp. | Top-20% |
|---|---|---|---|---|
| Termini-LogReg (appendix) | 980/980 | |||
| Termini-Chemprop (D-MPNN) | 951/980 | |||
| AMP-Modify-Eval | 980/980 |
Table 2. Task-matched comparison of pair-effect scores (single seed). The MIC-derived difference is the Spearman correlation between predicted and measured ; it is not a supervised pair-head score. NR means family-balanced Spearman was not computed for that score.
| Configuration/scoring method | Pair Sp. | Family-bal. Sp. | MIC- Sp. |
|---|---|---|---|
| Full model / supervised pair head | |||
| w/o pair loss / unsupervised pair head | — | ||
| w/o pair loss / MIC-derived difference | — | — |
Response to Reviewer nooZ, Q1
Q1. Question: Pair-label provenance and the C7 discrepancy. How are parent and modified-peptide MIC values matched before organism-group medians are formed—at the level of strain, study, and assay conditions—and how do results change on strictly matched pairs? In C7, the reported IDR1018 / PEG6-IDR1018 values (16 → 32 μg/mL; molecular weights 1535.8 / 1872.3 g/mol) convert to approximately 10.42 → 17.09 μM, a 1.64-fold increase rather than the two-fold molar increase required for “worsen” (Tables 2 and 8; Appendix C.1). What exact records justify that label? If these are the evaluated endpoints, please correct C7 and the nine-case summary. Please also quantify the effect of censored MIC substitutions on labels near the threshold.
Answer:
Thank you for pointing this out. Pair-effect labels are derived from matched experimental measurements, not differences between organism-group medians. For each parent--modified relation, we first match MIC records by species, strain, and activity-measurement group, requiring positive molar values from exact measurements or reported mean$\pm$SD. Among compatible records, the highest-scoring match is selected. Tier-A additionally requires the same publication and no conflicts in reported assay conditions (medium, CFU, pH, ionic strength, and salt type). The pair effect is then calculated directly from the matched μM values; organism-group medians are used only for the absolute MIC task. Our audit confirms that all 980 Tier-A pairs satisfy the strain/study matching criteria and have no recorded assay conflicts. Missing metadata are not treated as conflicts. 562 pairs have at least one unreported assay field. We therefore additionally evaluated the 418 pairs with all five context fields available. On this stricter subset, family-balanced Spearman is 0.331 versus 0.378 and Pair Spearman is 0.461 versus 0.403 (Table 1), using the same out-of-fold predictions without retraining. Pair-ranking therefore remains in a comparable range; we do not assume that unreported assay conditions are identical. Section 4.2 reports the 0.331 subset result, and the matching protocol is in the revised appendix.
We agree that the original worsen label is inconsistent with our molar MIC criterion. For the reported MRSA endpoint, IDR1018 and PEG6-IDR1018 have MICs of 16→32μg/mL and molecular weights of 1535.8 and 1872.3 g/mol, respectively. These correspond to 10.42→17.09μM, a 1.64-fold increase, which must be classified as neutral under our twofold threshold. We therefore relabel the reported C7 MRSA endpoint as neutral. The same publication reports 16→64μg/mL for PA14, E. coli, and A. baumannii, corresponding to a 3.28-fold molar increase and hence worsen; however, those distinct endpoints do not justify the original MRSA annotation. With the existing predictions unchanged, correcting C7 alone reduces our originally reported nine-case agreement from 7/9 to 6/9, while constant-improve remains at 7/9. Revised Table 2 and Section 4.3 use this corrected summary. The additional case-level corrections discussed in R1-Q8 are in the literature appendix.
None of the 980 Tier-A labels relies on censored MIC substitution. Both parent and modified measurements are exact or reported mean$\pm$SD values. The substitutions for censored measurements (<u↦u/2 and >u↦2u) are applied only when constructing the separate organism-group absolute-MIC data, where 16,789 of 60,551 records (27.7%) contain censored measurements. Consequently, censoring substitutions change 0/980 pair labels, including those near the decision boundary. Separately, the choice of direction threshold does affect label assignments. Changing the default twofold cutoff to 1.5-fold or fourfold changes 101 or 267 labels, respectively (Table 2). This sensitivity reflects the classification threshold, not censored-value imputation. The revised appendix now makes this distinction and the matching protocol explicit.
Table 1. Pair-ranking performance under stricter assay-context matching. Both subsets use the same single-seed out-of-fold predictions without retraining.
Table 2. Sensitivity of Tier-A direction labels to the fold-change threshold. Changed labels are counted relative to the default twofold cutoff; no Tier-A labels use censored MIC substitutions.