- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, PhyloAug, a method of using phylogenetic information to guide RNA data enhancement, is proposed. In this method, homologous sequences are retrieved and the relative evolution rate of each point is estimated, and the appropriate position is selected for Mask. Then, the structural conditional RNA language model is used to generate replacement bases and construct new fine-tuning data.
The author carried out experiments on seven RNA function prediction tasks and several models, and verified the effectiveness of the method through baseline comparison, ablation experiment, experiments with different labeled data and MMD distribution analysis. The results show that PhyloAug can improve the downstream prediction performance in most settings, but the effect depends on specific tasks and models.
Critical strengths
PhyloAug combines phylogenetic information with RNA structural condition generation, and can be applied to different RNA basic models without modifying downstream model architecture or training objectives.
The paper covers multiple tasks and models, and sets up baseline control and ablation experiments that match the training budget. It shows that the performance improvement is not only from the increase of training data, but also from the selection of phylogenetic sites.
Critical weaknesses
PhyloAug mainly combines the existing evolutionary rate estimation, data enhancement and structural condition generation techniques. EvoAug and PriFold have explored similar biological constraint enhancement ideas. In addition, the paper directly selected sites with relative evolution rate of [0.8, 1.2], but did not fully explain why these sites were more suitable for enhancement than highly conserved or rapidly evolving sites, and lacked corresponding systematic control experiments.
Methods It is assumed that the generated sequence can inherit the original function tag, but the evolution rate and structural conditions can not guarantee the function unchanged after mutation. Although the downstream performance and MMD results have improved, these evidences cannot directly prove that the generated sequence retains the original tag, thus weakening the biological rationality of the method.
Questions for authors
Q1. Why choose the evolution rate interval [0.8, 1.2]? Can we compare highly conservative, medium-speed and fast evolutionary sites under the same enhancement budget to verify the rationality of this choice?
Q2. Is there any direct evidence that the enhanced sequence can keep the original functional label? Can the existing experimental mutation effect data be used for verification?
Q3. Since the method has already retrieved homologous sequences, why not directly use the real homologous sequences screened by functional consistency for enhancement? Compared with simpler evolutionary heuristic enhancement strategies such as EvoAug, what is the additional benefit of PhyloAug?
4 comments
Reviewer 7JMW response Part II
Q2. Is there direct evidence of functional-label preservation, including from mutation-effect datasets?
We agree that downstream improvements and MMD measure different properties from direct functional-label preservation. To assess functional-label preservation, we added two task-specific analyses of label stability. For gRNA, whilst our generated variants did not have independent experimental measurements, we instead use the external DeepSpCas9 predictor to score original and augmented sequences, measuring Lin's concordance correlation coefficient (CCC) agreement. Agreement results were 0.707 for near-mean, 0.701 for accelerated and 0.667 for conserved selection. For m6A, we measured corruption of the DRACH motif associated with the labelled site. Motif-disruption rates were 5.2%, 5.4% and 10.5%, respectively, indicating that near-mean selection retained motif compatibility in 94.8% of evaluated variants. These analyses assess predicted activity stability and preservation of a recognised biological feature, rather than directly measuring variant function.
To quantify the performance gap associated with residual motif disruption, we additionally evaluated near-mean PhyloAug with enforced DRACH preservation. Across nine models, native PhyloAug improved mean F1 over NoAug by 0.0347, compared with 0.0441 for the motif-preserving reference: an absolute gap of 0.0094 F1. Native PhyloAug therefore achieved approximately 79% of the reference’s observed augmentation gain. The higher reference performance is consistent with a cost from residual motif disruption, while the positive native gain shows that this limitation does not negate the overall augmentation benefit. These results support inherited-label augmentation as a useful practical approximation, while explicitly quantifying a remaining limitation.
| Model | No-Aug | PhyloAug | DRACH-constrained |
|---|---|---|---|
| RNA-MSM | 0.6293$\pm$0.048 | +0.058$\pm$0.028 | +0.073$\pm$0.032 |
| SpliceBERT | 0.6634$\pm$0.006 | +0.037$\pm$0.012 | +0.044$\pm$0.015 |
| BiGamba | 0.6351$\pm$0.011 | +0.066$\pm$0.003 | +0.077$\pm$0.002 |
| RNA-FM | 0.6760$\pm$0.007 | +0.044$\pm$0.010 | +0.049$\pm$0.009 |
| RNAErnie | 0.6511$\pm$0.027 | +0.026$\pm$0.038 | +0.044$\pm$0.042 |
| HyenaDNA | 0.6845$\pm$0.021 | +0.034$\pm$0.011 | +0.037$\pm$0.012 |
| RNA-BERT | 0.5953$\pm$0.003 | +0.049$\pm$0.013 | +0.054$\pm$0.015 |
| CNN | 0.519$\pm$0.004 | -0.009$\pm$0.007 | +0.006$\pm$0.006 |
| ResNet | 0.529$\pm$0.006 | +0.007$\pm$0.006 | +0.013$\pm$0.001 |
| mean, all 9 | 0.6203 | +0.0347 | +0.0441 |
| mean, 7 pretrained | 0.6478 | +0.0449 | +0.0540 |
Q3. Why not augment with retrieved homologs, and what does PhyloAug add beyond EvoAug and PriFold?
Direct homolog augmentation is reasonable when matching task-specific labels are known; BLAST similarity alone does not establish this. PhyloAug instead uses homolog-derived rates to guide edits to labelled parents. We have not directly compared against functionally screened homologs or EvoAug on these tasks.
Table 1 isolates the added value of evolutionary selection: changing only the selector improves mean m6A F1 across 12 backbones by 0.0328 without structure conditioning and 0.0331 with it. Appendix J additionally compares PhyloAug with an offline PriFold-style operator at matched augmentation volume on three structure-prediction datasets, yielding 20 wins, 5 ties and 2 losses, with a mean advantage of 0.008 F1. This evaluates the augmentation operators, not PriFold’s complete framework or original online training schedule.
Response to rebuttal
Thanks to the authors for the detailed response and the extra experiments. However my main concerns aren't really resolved: label preservation is still only backed by proxy evidence, and there's still no comparison with EvoAug. I'll keep my score at 2.
Response to Reviewer 7JMW
Thank you for identifying the two remaining concerns. On label preservation, we agree that our evidence is indirect. We do not have independent experimental measurements for the generated variants. Retention of a label-associated motif and agreement with an external activity predictor are useful checks, but neither verifies that a variant retains its parent’s functional label. We will state this explicitly where label inheritance is introduced: inherited labels are a training assumption, and evaluation on unaugmented held-out examples measures predictive utility.
On EvoAug, we have now completed a direct comparison of augmentation operators on m6A prediction and CRISPR on-target activity, using nine backbones and five seeds per backbone. Both policies used the same parent sequences, augmentation counts per parent, inherited labels, downstream models and seeds, with the same caps on optimisation steps and tokens. We varied how the augmented sequences were generated.
The EvoAug-based arm used six discrete-sequence operators: deletion, insertion, translocation, inversion, mutation and reverse complementation. We excluded Gaussian noise because it acts on continuous one-hot inputs, and rescaled length-dependent operator ranges for the 23-nt CRISPR inputs. We applied both policies offline. This compares how the two augmentation policies perform within the same training protocol; it does not reproduce EvoAug’s full online augmentation and subsequent fine-tuning on unaugmented data. We will report the operator settings, sampling procedure and length rescaling in the appendix.
Across all nine models, PhyloAug had a mean advantage of 1.78 F1 percentage points on m6A and 0.0232 Spearman’s ρ on CRISPR, with higher mean scores on six and seven of the nine backbones, respectively. The model-level results are below.
m6A prediction: F1
| Model | NoAug, mean SD | PhyloAug, mean (score SD) | EvoAug-based, mean (score SD) |
|---|---|---|---|
| RNA-MSM | 0.6293 0.048 | +0.053 (0.038) | +0.068 (0.028) |
| SpliceBERT | 0.6634 0.006 | +0.041 (0.005) | +0.0007 (0.009) |
| BiGamba | 0.6351 0.011 | +0.063 (0.010) | +0.017 (0.005) |
| RNA-FM | 0.6760 0.007 | +0.050 (0.007) | +0.015 (0.013) |
| RNAErnie | 0.6511 0.027 | +0.027 (0.022) | +0.035 (0.025) |
| HyenaDNA | 0.6845 0.021 | +0.042 (0.013) | +0.007 (0.019) |
| RNA-BERT | 0.5953 0.026 | +0.031 (0.013) | 0.000 (0.082) |
| CNN | 0.5190 0.004 | +0.001 (0.004) | −0.002 (0.010) |
| ResNet | 0.5290 0.006 | −0.003 (0.006) | +0.004 (0.010) |
| Mean, all 9 | 0.6203 | +0.0339 | +0.0161 |
| Mean, 7 pretrained | 0.6478 | +0.0439 | +0.0204 |
CRISPR on-target activity: Spearman’s ρ
| Model | NoAug, mean SD | PhyloAug, mean (score SD) | EvoAug-based, mean (score SD) |
|---|---|---|---|
| HyenaDNA | 0.3184 0.010 | +0.0295 (0.010) | +0.0076 (0.051) |
| SpliceBERT | 0.3198 0.030 | +0.0082 (0.030) | +0.0046 (0.031) |
| RNAErnie | 0.3566 0.010 | +0.0024 (0.010) | +0.0070 (0.051) |
| RNA-FM | 0.3646 0.020 | +0.0364 (0.020) | −0.0169 (0.041) |
| BiGamba | 0.1779 0.030 | +0.0060 (0.020) | +0.0011 (0.045) |
| RNA-MSM | 0.3252 0.020 | +0.0268 (0.020) | −0.0151 (0.027) |
| RNA-BERT | 0.2322 0.020 | +0.0568 (0.020) | +0.0204 (0.037) |
| ResNet | 0.2130 0.010 | +0.0290 (0.030) | −0.0243 (0.070) |
| CNN | 0.1600 0.020 | −0.0260 (0.010) | −0.0242 (0.030) |
| Mean, all 9 | 0.2742 | +0.0188 | −0.0044 |
| Mean, 7 pretrained | 0.2992 | +0.0237 | +0.0012 |
For both tables, mean is the change in mean held-out score relative to NoAug for the same backbone. Parentheses give the across-seed SD of the augmented model’s scores, not the SD of paired gains. Bold marks the higher mean of the two augmentation policies.
The EvoAug-based policy improved mean m6A F1 over NoAug and performed better than PhyloAug on some backbones. The evidence therefore supports an average predictive advantage for PhyloAug on these two tasks under the matched offline protocol. A separate m6A control helps identify one contributor to that advantage over random editing: with DRACH protected in both arms and the generator held fixed, phylogenetic site selection outperformed random site selection by 2.45 F1 percentage points on average across nine models. This shows that edit-site choice matters even after explicit motif protection. Whether individual variants retain their biological labels remains an open experimental question.
Reviewer 7JMW response Part I
We thank the reviewer for their thoughtful feedback and apologise for splitting the response to meet the character limit.
Q1. Why choose [0.8, 1.2], and how does it compare with conserved or rapidly evolving sites?
We agree that the rate-selection rule requires a direct comparison with alternative evolutionary regimes. The interval [0.8, 1.2] is a fixed heuristic around the normalised mean evolutionary rate of one; it was not selected using downstream validation performance and does not itself establish selective neutrality. To test this choice, we added a matched-budget comparison of conserved, near-mean and accelerated-site perturbations on m6A and gRNA, covering motif-driven and sequence-to-function prediction. We define conserved sites as r≤0.8, near-mean sites as 0.8≤r≤1.2, and high-rate sites as r≥1.2. The near-mean interval is a fixed heuristic around the normalised alignment mean of one, not a validation-selected interval. Across nine backbones and five seeds, the comparisons use matched parents, mask counts, generation settings and training protocols, with identical augmentation-budget searches.
Our matched-budget comparison shows mean m6A F1 changes over NoAug of +0.0347 for near-mean, +0.0303 for high-rate and −0.0003 for low-rate editing. The clearest separation is therefore from low-rate editing; the additional benefit over high-rate editing is smaller. Corresponding gRNA Spearman changes are +0.0055, −0.0002 and −0.0221. These results support the proposed interval as a useful default, with a modest observed advantage over accelerated-site editing and a clearer advantage over conserved-site editing. They support the choice of rate regime without establishing that the exact endpoints are uniquely optimal. We will add these experiments to the manuscript.
Experiment 1 m6A Result:
Experiment 1 gRNA Result: