- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents PLM-CARE, a framework for assessing and reducing predicted virulence-factor (VF) association in protein sequences generated by protein language models. The framework combines evolutionary-context generation, multiple VF predictors, and quality-matched preference alignment. VirulentHunter is used to construct preference pairs, while a DPO-based objective is adapted to the masked generation of ESM3 and the autoregressive generation of PoET-2.
Experiments on held-out evolutionary contexts show reductions in VF-associated predictions from two external classifiers, DTVF and VirulentPred 2.0, after fine-tuning. The authors also evaluate sequence validity, evolutionary-profile compatibility, and other sequence-quality proxies, reporting no observed degradation in these metrics.
Critical strengths
1.The paper evaluates both ESM3 and PoET-2, covering masked and autoregressive generation. The consistent reductions in predicted VF association across the two models suggest that the proposed alignment approach is not limited to a single generation architecture.
2.The authors use separate classifiers for preference construction and evaluation, report results across multiple training seeds, and examine several sequence-quality metrics. The homology-aware data splitting and ablation experiments also strengthen the experimental analysis.
Critical weaknesses
-
The proposed framework mainly combines existing VF predictors, preference filtering, and DPO-based fine-tuning. While applying the framework to different PLM architectures is useful, the methodological contribution appears relatively limited. It is also unclear whether the proposed quality-matched preference construction offers substantial benefits over standard preference optimization.
-
The evaluation relies entirely on predicted VF-associated scores, without direct biological validation. Although multiple classifiers are considered, their agreement at the prompt level is relatively weak. This raises concerns about whether the observed reductions reflect meaningful changes in biological properties or simply differences in the sequence features captured by these predictors.
-
The experiments focus on two PLMs and a relatively narrow set of risk-enriched evolutionary contexts. It remains unclear whether the method generalizes to other protein families or generation settings. In addition, the reported quality metrics mainly assess sequence validity and evolutionary compatibility, which do not necessarily indicate preserved biological function.
Questions for authors
-
Section 4.5 includes an ablation that removes the quality-matching constraint, but the reported results mainly focus on sequence-quality metrics. Could the authors also report the corresponding changes in VF-associated predictions for this ablation? This would help clarify whether quality matching provides a better trade-off between reducing VF-associated signals and preserving sequence quality. Comparisons with simpler fine-tuning baselines would further help establish the contribution of QMPA.
-
The paper reports consistent overall reductions across external classifiers, but the prompt-level effect correlations are relatively weak, particularly for PoET-2. Could the authors further analyze these disagreements and discuss whether the alignment primarily optimizes features associated with the preference-labeling classifier rather than producing consistent changes in biologically meaningful VF-associated properties?
-
While the paper reports sequence uniqueness and embedding-space visualizations, these analyses do not fully characterize the diversity or structural quality of generated proteins. Could the authors provide additional evaluations, such as pairwise sequence identity and predicted structural confidence? It would also be helpful to assess whether fine-tuning preserves the predicted functional properties of generated sequences.
-
Both models are evaluated primarily on risk-enriched evolutionary contexts derived from VF-associated proteins. How well does the alignment generalize to other protein families or generation conditions? In particular, does the method affect the generation quality or functional predictions of proteins that are not associated with virulence factors?
9 comments
Response to Reviewer 8chY — Q1 (2/2)
In addition to the fine-tuning baseline without the quality-matching constraint, we added a supplementary chosen-only SFT experiment as a simpler baseline. This addresses the concern in your critical weaknesses that the proposed quality-matched preference construction may offer limited advantages over standard preference optimization. The results show that QMPA also has advantages over chosen-only SFT. We trained SFT using only the preferred sequence from each QMPA pair and evaluated both methods on the same held-out contexts. The table below presents the results.
Comparison of QMPA and chosen-only SFT
| Architecture | Method | Mean DTVF | ΔDTVF vs. Base (95% CI) | Mean Q_profile | ΔQ_profile vs. Base (95% CI where available) |
|---|---|---|---|---|---|
| ESM3 | QMPA | 0.5166 | −0.1491 [−0.2510, −0.0731] | 48.4256 | +0.9564 [0.2455, 1.7116] |
| ESM3 | Chosen-only SFT | 0.6478 | −0.0179 [−0.0353, −0.0052] | 47.1185 | −0.3507 [−0.5221, −0.2129] |
| ESM3 | Base | 0.6656 | 0 | 47.4692 | 0 |
| PoET-2 | QMPA | 0.7351 | −0.1405 [−0.1819, −0.0955] | 40.0968 | +10.6034 [9.3642, 11.8480] |
| PoET-2 | Chosen-only SFT | 0.9540 | +0.0784 [0.0673, 0.0900] | 22.9128 | −6.5806 |
| PoET-2 | Base | 0.8756 | 0 | 29.4934 | 0 |
Direct comparison of QMPA and chosen-only SFT
| Architecture | ΔDTVF | ΔQ_profile | ΔVirulentHunter |
|---|---|---|---|
| ESM3 | −0.1312 [−0.2173, −0.0657] | +1.3071 [0.6155, 2.1325] | −0.1894 [−0.2406, −0.1450] |
| PoET-2 | −0.2189 [−0.2424, −0.1965] | +17.1840 [15.6148, 18.7695] | +0.0564 |
For PoET-2, QMPA changed DTVF by −0.2189 and Q_profile by +17.1840 relative to SFT. The corresponding ESM3 analysis also favored QMPA on DTVF (−0.1312) and Q_profile (+1.3071).
These supplementary experiments demonstrate advantages for QMPA over baseline preference optimization.
In Q1, we added both a joint analysis of the quality-matching ablation and VF prediction scores and a chosen-only SFT control. We will replace the original quality ablation in Section 4.5 with the new quality analysis. We will also report the scores from the external evaluators and the preference-labeling classifier to extend the conclusions of the ablation. The chosen-only SFT control demonstrates QMPA's added value over simple fine-tuning, and we will add it to the ablation experiments in Section 4.5.
Response to Reviewer 8chY — Q2
We acknowledge the difference between the aggregate trends and the per-context results. To analyze these discrepancies, we conducted a supplementary paired analysis of cross-evaluator agreement on common contexts. For each of the 137 PoET-2 contexts that both DTVF and VP2 could score, we computed the change from Base to QMPA. On the 137 shared PoET-2 contexts, both external evaluators showed mean decreases after QMPA. Agreement on which individual contexts improved was only slightly above that expected from the marginal decrease rates (77.4% observed vs. 76.0% expected). These results support an aggregate effect across the two external predictors,We then compared how closely the classifiers agreed on the direction of the changes and on their ranking by magnitude. The table below presents the results.
Agreement between external evaluators on common PoET-2 contexts
| Measure | Result |
|---|---|
| Mean DTVF change | −0.1722 |
| Mean VP2 positive-rate change | −18.89 percentage points |
| Both endpoints decreased | 103/137 |
| Only one endpoint decreased | 31/137 |
| Neither endpoint decreased | 3/137 |
| Observed agreement in direction | 77.4% |
| Expected agreement from marginal decrease rates | 76.0% |
| Cohen's κ | 0.056 |
| Spearman's ρ for the changes (95% CI) | 0.023 [−0.264, 0.305] |
The results show that DTVF fell by 0.1722 on average, the VP2 positive rate fell by 18.89 percentage points on average, and both measures fell in 103 contexts. Each measure fell in most contexts, so the expected agreement based on their marginal decrease rates alone was 76.0%, close to the observed agreement of 77.4%. The corresponding Cohen's κ was 0.056. The Spearman correlation for the magnitude of the changes was 0.023, showing little agreement in the rankings. Thus, both classifiers showed an average decrease across these contexts, while agreement beyond chance on which contexts improved was weak, as was agreement on which contexts improved the most.
To understand this pattern, we examined the prediction tasks and modeling approaches of the two evaluators in more detail. Both DTVF and VP2 are used to identify virulence-associated proteins. DTVF combines representations from the ProtT5 protein language model with a deep learning model to make predictions, while VP2 uses position-specific scoring matrices (PSSMs) as features for classification. Their prediction objectives are similar, but they extract and use different sequence information and may therefore respond differently to changes in the same set of sequences.
In addition, this analysis compared changes in continuous DTVF scores with changes in VP2 positive rates. A continuous score can change while a sequence stays on the same side of the positive classification threshold. Changes in the positive rate primarily reflect the fraction of samples crossing that threshold. The two statistics may therefore rank the magnitudes of change differently. The differences in models and statistics provide plausible explanations for the low correlation, although this agreement analysis leaves the contribution of each factor unresolved.
You asked whether our fine-tuning method simply teaches the model to generate “sequences that this classifier will score lower,” and whether these changes correspond to real, biologically meaningful reductions in virulence. Our evaluation was designed to address precisely this question. We use VirulentHunter to construct the preferred and rejected sequences, and , and use DTVF and VP2 for evaluation, providing an analysis based on biological predictors. We use different predictors during training and testing. VirulentHunter is used only during training, and DTVF and VP2 are used only during testing. The test results show that the improvements obtained with VirulentHunter-labeled preference pairs transfer to the external evaluators. VirulentHunter is a leading protein virulence-factor predictor with a strong design. Using it as an optimization surrogate therefore helped the model learn biologically relevant preferences efficiently.
The supplementary experiment in this response clarifies what we mean when we say that the external evaluators corroborate each other. We will add the experiment and its main conclusions to Section 4.2, expanding the original comparison between evaluators.
Response to Reviewer 8chY — Q1 (1/2)
Thank you for your constructive feedback and for highlighting the strengths of our cross-architecture evaluation.
The original quality-matching ablation focused on the direct objective of the constraint. It tested whether introducing quality matching during preference optimization helps preserve the quality proxy. We therefore first reported the change in Q_profile for QMPA relative to noQ. Your suggestion makes the comparison more complete and helps us assess how quality matching relates to improvements in VF-associated scores.
We therefore expanded the quality-matching ablation. First, we evaluated Base, QMPA, and noQ on a common cohort to make the ablation more rigorous. In addition to changes in Q_profile, we compared predictions for Base, QMPA, and noQ from the external evaluators DTVF and VirulentPred 2.0 (VP2). ESM3 VP2 is reported on the 129 contexts scored in all three arms.
Quality-matching ablation on a common cohort
Comparing Q_profile for QMPA and noQ on the same cohort for ESM3 and PoET-2 yields the same conclusion as the original manuscript. Removing the quality-matching constraint lowers the fine-tuned models' generation quality as measured by Q_profile relative to QMPA.
To answer your follow-up question, we first scored all three arms with the two external evaluators. For ESM3, both evaluators gave QMPA lower mean risk scores than noQ, with the same ordering of means in both cases (QMPA < noQ < Base). The pattern was different for PoET-2. QMPA again received lower risk scores, while noQ's DTVF score increased relative to Base. This was unexpected, so we also computed VirulentHunter (VH) scores for all three PoET-2 arms. We found that noQ achieved the greatest improvement on VirulentHunter, the classifier used to construct the preferences, but that advantage did not carry over to the external evaluators. This suggests that removing the quality-matching constraint may lead fine-tuning to cater to the classifier's preferences at the expense of learning genuine biological properties. We address this point further in our response to Q2.
We likewise computed the classifier scores for all three ESM3 methods. The VirulentHunter scores followed the same trend as the external evaluator scores.
The additional risk scores show why VirulentHunter, the classifier used during training, is insufficient on its own to judge the effect of fine-tuning. They also show that MVA's separation of the labeling classifier from the external evaluators gives a fuller picture of where fine-tuning has an effect. Finally, they show that QMPA can retain an improvement on VirulentHunter while transferring that improvement to external evaluators, yielding more consistent improvements in the average external scores.