- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a multi-attribute controllable protein sequence generation framework based on a general language model - Aria. The authors fine-tuned Llama-3.1-8B-Instruct using QLoRA and the training data came from approximately 349,000 Swiss-Prot protein sequences. This model demonstrated significant improvement in the adherence to sequence-level attributes (such as length, hydrophobicity, cysteine count, transmembrane topology, and signal peptide). However, in extensive evaluations, the ability to recover the required Pfam domains was still limited.The authors also introduced a DPO method based on natural sequences, that is, using natural protein sequences as the preferred response and rejecting the generated sequences with low confidence. In experiments on multiple protein families, the results showed that the confidence of ESMfold structure prediction, the recovery rate of Pfam domains, and the predicted structural similarity all improved. However, these improvements mainly relied on specific protein families and there is no sufficient evidence to show that they have effective transferability across families.
Critical strengths
1.Systematic assessment of the controllability of multi-attribute proteins.This paper conducts a comprehensive experimental analysis of controllability at various levels, including sequence composition, topological structure, and domain identity. By using grouped separated evaluation data, rapid misalignment control, and multiple specific attribute indicators, the empirical results are further strengthened. The attributes that can be directly calculated from the sequence and those that require external predictors are distinguished, indicating that after the specific instructions of proteins are optimized, general-purpose language models can or cannot reliably control certain aspects.
- Significant empirical improvement and detailed analysis of preference optimization.The preference optimization experiment based on natural sequences shows that in multiple protein families, the prediction of structural confidence and the recovery rate of the requested domains have both improved.
Critical weaknesses
-
Limited methodological novelty and insufficient baselines.The framework primarily combines existing instruction tuning and DPO techniques. Comparisons with the untuned Llama model and ProLLaMA are insufficient, as ProLLaMA cannot receive the same property specifications. Stronger baselines under matched conditioning settings are needed.
-
Insufficient DPO ablations and limited generalization.The improvements from anchored DPO may partly result from additional supervised learning on natural sequences. Comparisons with continued SFT or NLL-only training are missing. Moreover, overall Pfam recovery remains low (2.9%), and family-specific gains transfer poorly, raising concerns about generalizability.
Questions for authors
1.The comparison with ProLLaMA is somewhat difficult to interpret, since ProLLaMA does not receive the same property specifications as Aria. Could the authors include a protein-specific baseline adapted to the same conditioning inputs? Alternatively, a retrieval-based or constraint-filtering baseline would help establish how much improvement comes from the proposed approach. 2. I am not fully convinced that the improvements after anchored DPO are primarily due to preference optimization. Since natural protein sequences are used as preferred responses and an additional NLL loss is included, some gains might simply come from further supervised training. A comparison with continued SFT under a similar training budget would help clarify this point. 3. While the improvements in individual properties are encouraging, it remains unclear how often the generated proteins satisfy multiple constraints simultaneously. In particular, the strict composite metric does not include Pfam-domain recovery. It would be useful to report joint success rates incorporating domain identity and structural properties, especially for protein families with limited representation in the training data.
Response to question 2
We thank the reviewer for suggesting this control. We agree that variability between structure predictors can affect the results. As per reviewer’s suggestion, for both families, uL24 (PF00467) and YqgF (PF03652), we used OmegaFold to predict structures for the same generated sequences and their accession-matched natural references. We then repeated the TM-score comparison using the same alignment procedure as for ESMFold. At the round-three DPO checkpoint, both mean predicted confidence and mean TM-score increased relative to supervised Aria in both families at both seeds.
For uL24, the OmegaFold pLDDT gains were 9.2 and 5.3 points at the two seeds, and the corresponding TM-score gains were 0.294 and 0.250. For YqgF, pLDDT increased by 14.6 and 13.7 points, and TM-score increased by 0.156 and 0.167. The TM-score gains were close to those obtained with ESMFold on the same generations: 0.290 and 0.242 for uL24, and 0.153 and 0.185 for YqgF. The matched continued-SFT checkpoints also improved both OmegaFold measures in both families at both seeds. We added a section to the appendix (Section M) to report these findings.