- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UNAAGI, an atom-level diffusion framework for generating canonical and non-canonical amino acid (NCAA) substitutions and predicting their mutational effects. Unlike conventional inverse-folding models that predict residue identities from a fixed vocabulary, UNAAGI uses an E(3)-equivariant diffusion model to generate side-chain molecular graphs conditioned on local protein structures. The generated graphs are mapped to amino acid identities, and their sampling frequencies are used to estimate mutation likelihoods and derive zero-shot mutational-effect scores.
Critical strengths
Replacing fixed residue identities with atom-level molecular graph generation is a reasonable and interesting approach to non-canonical amino acid design. By directly modeling atomic composition, chemical bonds, and 3D conformations, UNAAGI can represent a broader range of side-chain chemistries without requiring a predefined output class for every amino acid. This provides a natural connection between molecular generative modeling and protein inverse folding, and offers a promising direction for extending protein design beyond the canonical amino acid vocabulary.
Critical weaknesses
-
The main motivation of UNAAGI is to overcome the fixed vocabulary of conventional inverse-folding models. However, after generating atomic-level molecular graphs, the authors still map them to amino acid identities using a reference library. This raises a question about whether atom-level diffusion is really necessary for the evaluated task. For example, could a conventional inverse-folding model simply extend its output vocabulary to include the NCAAs in the reference library? While UNAAGI can theoretically generate molecular structures outside this library, the paper does not clearly demonstrate the practical advantage of this capability. Also, the model is already trained on substantial NCAA structural data, including isolated NCAA molecules and NCAA-containing protein structures. Therefore, its ability to generate NCAAs does not necessarily demonstrate generalization to previously unseen amino acids. Although the authors conduct held-out experiments, they only exclude four canonical amino acids rather than actual NCAAs. Moreover, these experiments report coverage, which only measures whether an amino acid appears at least once among many generated samples. With 1,000 samples, even an amino acid assigned a probability of just 0.5% has over a 99% chance of appearing at least once. Thus, high coverage does not necessarily indicate that the model has learned accurate residue probabilities or meaningful preferences.
-
The main experimental results show positive Spearman correlations between UNAAGI's mutation scores and experimental measurements. However, it is unclear whether these correlations reflect the model's ability to distinguish different NCAA substitutions or simply its ability to identify positions that are generally sensitive to mutations. For example, consider a position where the model frequently generates the wild-type amino acid but never generates any of the tested NCAAs. Because the authors assign the same pseudocount to all unsampled amino acids, every NCAA substitution at this position receives exactly the same mutation score. The model therefore cannot distinguish which NCAA would be better or worse at that position. However, different positions may have different wild-type probabilities. If the model can identify which positions are generally more tolerant of mutations, it could still achieve a positive Spearman correlation when all substitutions across all positions are evaluated together. In other words, a positive overall correlation does not necessarily demonstrate that the model has learned the chemical differences between NCAA substitutions. I think this is particularly important because the paper's central goal is to extend mutational-effect prediction beyond canonical amino acids. The authors should evaluate whether UNAAGI can rank different NCAA substitutions at the same position, and compare against simple baselines that only capture position-level mutation sensitivity. In addition, the training-data ablations are only evaluated on canonical ProteinGym benchmarks, rather than the NCAA benchmarks that form the main contribution. Finally, estimating mutation probabilities requires 1,000 diffusion samples per position, each involving 20 denoising steps. Given this substantial computational cost, the paper would benefit from runtime comparisons and stronger evidence that atom-level diffusion provides advantages over simpler prediction methods.
Questions for authors
- Have the authors considered a leave-one-NCAA-out experiment to directly evaluate generalization to unseen amino acid chemistries? Additionally, given that generated molecular graphs are ultimately mapped to a reference library, what advantages does UNAAGI offer over simply extending the vocabulary of a conventional inverse-folding model?
- Could the authors clarify whether the reported Spearman correlations reflect the ability to distinguish different NCAA substitutions at the same position, rather than simply identifying positions that are generally sensitive to mutations? In particular, how frequently are NCAA substitutions never sampled and therefore assigned identical pseudocount-based scores? A within-site ranking evaluation or a simple site-level baseline would help establish whether the model actually captures NCAA-specific mutational effects.
4 comments
Response to Reviewer EZdD (2/2)
5. Ablations on the NCAA benchmarks (W2)
To assess the contribution of each training source, we evaluated the ablated models on CP2 and PUMA over five sampling runs. The table reports Spearman’s ρ for all evaluated substitutions and for the NCAA-only subset. Table 5 of the revised manuscript is reproduced below.
| Training configuration | CP2, all | CP2, NCAA only | PUMA, all | PUMA, NCAA only |
|---|---|---|---|---|
| Full UNAAGI | 0.3241 ± 0.022 | 0.4283 ± 0.009 | 0.3455 ± 0.027 | 0.3216 ± 0.029 |
| Without NCAA-containing PDB data | 0.2244 ± 0.052 | 0.3298 ± 0.053 | 0.2729 ± 0.040 | 0.2297 ± 0.032 |
| Without PDBBind | 0.2814 ± 0.048 | 0.3795 ± 0.055 | 0.3235 ± 0.047 | 0.3195 ± 0.043 |
| Without isolated-NCAA data | 0.2402 ± 0.055 | 0.3577 ± 0.050 | 0.2597 ± 0.028 | 0.2399 ± 0.035 |
| Canonical PDB data only | 0.1851 ± 0.068 | 0.2589 ± 0.074 | 0.2132 ± 0.032 | 0.2244 ± 0.033 |
6. Sensitivity to the pseudocount (Q2)
Before computing ΔNLL, we add the same pseudocount k to every residue count. Table 4 reports Spearman correlations on the full evaluation sets for five values of k and includes a separate analysis restricted to substitutions sampled by UNAAGI.
| Pseudocount k or evaluation subset | CP2, all | CP2, NCAA only | PUMA, all | PUMA, NCAA only |
|---|---|---|---|---|
| 0.1 | 0.2697 ± 0.029 | 0.3891 ± 0.009 | 0.2890 ± 0.025 | 0.2941 ± 0.032 |
| 1.0 (reported) | 0.3241 ± 0.022 | 0.4283 ± 0.009 | 0.3455 ± 0.027 | 0.3216 ± 0.029 |
| 5.0 | 0.3492 ± 0.018 | 0.4400 ± 0.010 | 0.3717 ± 0.029 | 0.3301 ± 0.028 |
| 10.0 | 0.3536 ± 0.017 | 0.4396 ± 0.009 | 0.3766 ± 0.030 | 0.3295 ± 0.029 |
| 20.0 | 0.3528 ± 0.019 | 0.4384 ± 0.010 | 0.3696 ± 0.030 | 0.3293 ± 0.030 |
| Sampled substitutions only | 0.3300 ± 0.037 | 0.4712 ± 0.047 | 0.3655 ± 0.024 | 0.3035 ± 0.057 |
Thank you for the detailed response and additional experiments. I appreciate the new within-position analysis and NCAA-specific ablations, which address several of my concerns.
However, I am still not fully convinced of the necessity of atom-level diffusion. The authors argue that extending the vocabulary of conventional inverse-folding models would suffer from data sparsity, while atom-level generation enables knowledge sharing across different NCAAs. Although reasonable, this argument remains largely conceptual and is not supported by direct comparisons. Simpler alternatives, such as extended-vocabulary models or structure-conditioned predictors using molecular graph embeddings, could potentially achieve similar benefits.
Moreover, the substantial inference cost of estimating mutation probabilities through 1,000 diffusion samples per position remains unaddressed. Without comparisons against simpler NCAA-aware models or an analysis of the sampling budget, it is difficult to assess whether the additional complexity of atom-level diffusion is justified.
Dear reviewer EZdD,
Thanks for your comments.
We just want to clarify one important aspect of our paper, which has perhaps not been sufficiently clear: We agree with the reviewer that extended-vocabulary models might work well in principle. But in practice, we do not have the data to train them. As illustrated in our new Table 3, many of the amino acids that are experimentally interesting do not appear a single time in the PDB. If we train an inverse folding model with an extended vocubulary, there would therefore be no signal to update the logit-dimensions corresponding to these NCAAs. The mixing-in of isolated NCAAs works in our model because transfer of information between different amino acids is possible, but in an extended-vocabulary model, there is nothing that would couple NCAAs to their closest canonical AAs. To clarify this point in the paper, we now included a discussion on the contrast to extended vocabulary methods in Section F.1 of the paper.
We agree that training a structure-conditional model to predict small molecule embeddings is another potential solution to this problem. Actually, we followed this approach in parallel to the diffusion model presented here - but were never able to get competitive performance out of that approach. We agree that it would be reasonable to further investigate the potential of structure conditioned embedding-prediction in the future, but we believe that this is out of scope for this current paper.
Our main objective with this paper was to demonstrate that generalization from canonical to nearby non-canonical amino acids was possible, and that this can lead to useful levels of correlation to experimental data. We agree that further gains can probably be achieved in this area - but as we demonstrate in the paper, no other methods can currently explore NCAA space reliably, and we therefore believe that already this current version has real value for peptide exploration in a pharmaceutical context.
Response to Reviewer EZdD (1/2)
We thank the reviewer for asking what atom-level generation adds and whether UNAAGI can distinguish NCAA substitutions at the same site. These questions helped us clarify the rationale for our approach and strengthen its evaluation. We report within-position correlations, quantify overlap between the training data and the benchmarks, and evaluate the contribution of each training source on CP2 and PUMA.
1. Ranking alternative substitutions at the same position (W2, Q2)
To assess whether UNAAGI can distinguish among NCAA substitutions at the same site, we added a within-position analysis in Figure D3 of the revised manuscript. At each position, we compute Spearman’s ρ between predicted scores and measured mutational effects, then average the correlations across positions.
At a given position, the wild-type term in ΔNLL is shared by all candidates and therefore leaves their ranking unchanged. The positive mean NCAA-only correlations indicate that UNAAGI captures differences in mutational effects among NCAA substitutions at the same site. Figure D4 complements this analysis by comparing predicted and measured effects for each residue identity across positions, where the wild-type term can vary.
In Figure D2, cells in the NCAA columns mark substitutions that UNAAGI sampled at least once at the corresponding position in each of the five runs.
2. Why atom-level generation for open-ended NCAA design? (W1, Q1)
UNAAGI generates side-chain atoms, bonds, and three-dimensional coordinates conditioned on the local protein environment. This allows it to propose NCAA structures beyond the benchmark panel. Section 4.7 and Figure E1 illustrate the range of sampled chemistries. We use the reference library after generation to identify sampled residues and estimate their frequencies for substitutions with experimental measurements.
Extending a residue-token model such as ProteinMPNN introduces a separate output class for each added NCAA, many of which have few training examples. UNAAGI shares atom and bond representations across residue identities, allowing examples of one side chain to inform predictions for chemically related NCAAs. The same representation can express new combinations of atoms and bonds as the set of candidate chemistries grows. We clarify this rationale in Appendix F of the revised manuscript.
3. Interpretation of held-out recovery (W1)
We count a site as covered if UNAAGI generates the held-out canonical residue at least once in each of five runs of 1,000 draws. Recovery rates were significantly higher at experimentally favorable sites than at harmful sites (Table 1 and Appendix D.1). This suggests that UNAAGI can recover site-specific preferences for a residue identity withheld during training.
4. Training overlap (W1, Q1)
Table 3 of the revised manuscript, reproduced below, reports how many records in each of the two NCAA-specific training sources match the heavy-atom molecular graph of each audited NCAA.
Across the two NCAA-specific training sources, 14 of the 20 audited residue graphs have a match. The six unmatched graphs are 2TH, 3TH, AHP, BZT, TBU, and TME. Our PDBBind training examples contain small-molecule ligands and provide no additional NCAA residue examples. Figure D2 shows coverage at each position in CP2 and PUMA, including positions where UNAAGI generates AHP and TBU despite their absence as NCAA training examples. Figure D4 includes across-position correlations for both AHP and TBU.