- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces IDiom, an autoregressive protein language model trained on IDiom-DB, a collection of 54 million predicted intrinsically disordered regions curated from the AlphaFold Database. A fill-in-the-middle training format supports both unconditional IDR generation and generation conditioned on flanking protein sequences. The authors characterize the resulting sequences using sequence similarity, amino acid composition, patterning statistics, and several disorder predictors.The main methodological contribution is RL-SAE: the authors train a sparse autoencoder on IDiom representations, identify features enriched in function-associated sequence sets, and use reinforcement learning to reward the simultaneous activation of selected features. The reward also incorporates length and compositional-entropy penalties, with KL regularization to the base model. Importantly, reward features are evaluated using the frozen pretrained model and SAE. Experiments cover six compartment-association targets, transcriptional activation and repression domains, and combinations of chromosome-association and effector features. Comparisons include activation steering, supervised fine-tuning, and direct ProtGPS-reward optimization. The reported results support improved target-feature coverage and several improved computational readouts, while biological functionality of the generated sequences remains experimentally unvalidated.
Critical strengths
- The paper provides a concrete and effective mechanism for controlling multiple learned sequence features. RL-SAE turns a selected set of SAE features into an explicit sequence-level objective, rather than assuming that a sum of steering directions will successfully install all corresponding features. The reported average coverage of approximately 90% of the targeted top-30 features, compared with 24% for activation steering, is a substantial difference on the stated control objective. Evaluating rewards through a frozen pretrained model and SAE also keeps the feature definition fixed throughout optimization
- The feature-selection and composition experiments go beyond simply improving a single predictor score. The private-feature experiment provides an informative intervention: removing shared features reduces particular off-target predictions while largely preserving the intended target prediction. For example, the chromosome-targeted designs’ nucleolus score decreases from 0.85 to 0.27, while their chromosome score changes from 0.71 to 0.67. The chromosome-plus-effector experiments further show that the model can install approximately 53 of 60 requested features and organize incompatible acidic and basic features into different sequence regions without an explicit positional specification.
Critical weaknesses
- The described splitting and deduplication procedures do not clearly establish independent evaluation of the pretrained model. Appendix A describes duplicating the 54 million IDRs into prompted and unprompted examples, producing 108 million sequences, and then splitting these into training, validation, and test sets. Appendix B explicitly describes a random split of these examples. As written, this could place the prompted version of an IDR in training and its unprompted counterpart in validation or test. The paper does not state that these paired examples are assigned to the same partition.
- The evidence is stronger for surrogate-feature control than for reliable, function-specific biological design. The authors appropriately acknowledge that the compartment-associated sequences used for feature enrichment come from the same full-length proteins used to train ProtGPS. Thus, the main localization evaluation is not independent of the information used to define the design targets. DeepLoc is a useful additional readout, but the nuclear-versus-cytoplasmic evaluation cannot establish discrimination between the finer compartments central to the paper. Moreover, the compartment labels originate from full-length proteins, whereas the generated objects are isolated IDRs. The presented evidence does not establish that all extracted regions independently determine the corresponding localization.
Questions for authors
- Data Partitioning Are the prompt and non-prompt versions of each IDR assigned to the same partition? Are all IDRs of the same parent protein and their surrounding contexts also kept together? Please specify the actual partitioning unit and provide the number of overlapping focal IDRs and parent proteins in the training set, validation set, and test set.
- DisProt De-duplication Issue Does the de-duplication check in DisProt only target the core IDR in each training record, or does it also include all retained N-terminal and C-terminal contexts? In particular, could another retained IDR in the same parent protein potentially expose a benchmark region during pre-training? Review this possibility and determine which reported evaluation results will be affected and significantly influence my judgment if such overlaps exist.
- Function-specificity In addition to the output results of ProtGPS and coarse-grained DeepLoc, what evidence supports the fine specificity of the selected features on subcellular compartments? For example, can these features distinguish the IDRs of the target compartment from those of other human compartments with matching length and composition on the retained parental proteins? Conduct targeted analysis on such questions to help determine whether the selected features capture the compartment-specific information rather than broader sequence features.
2 comments
Thank you for your reply. I believe the new analysis has enhanced the credibility of the paper and has addressed my main concerns within a reasonable range. Therefore, I have accordingly raised my score.
We thank Reviewer 1i8H for their helpful assessment of the manuscript and constructive comments. We have uploaded a new version of the manuscript.
Yes, prompted and unprompted versions of each IDR are assigned to the same partition. We split IDR records into the different partitions before training, and we now clarify this in Appendix A.
The IDRs of the same parent protein are not kept together, and data splitting was only done according to the focal IDR. We have screened for the number of focal IDRs in the validation and test sets which are found anywhere within the full sequence of any training record. We find that 147,915 of 270,775 validation IDRs (54%) and 147,682 of 270,777 test IDRs (54%) show up within a training record. The number of parents shared between training-validation is 139,863 and the number shared between training-test is 139,673.
We now remove from the validation and test sets those above records where the focal IDR is found anywhere within a training record. We then re-evaluate the model validation perplexity (Figure 1d), finding that for the 300M parameter model, unprompted IDR validation perplexity increases from 10.81 to 11.85, and prompted IDR perplexity increases from 9.58 to 10.82.
DisProt deduplication was also only applied to the focal IDR of each training record. We find that 913 of 1,663 DisProt evaluation focal IDRs (55%) are found anywhere within a training record, and we now remove those IDRs from the DisProt evaluation set. For the 750 remaining, we re-calculate model perplexities (Figure 1d), finding that the 300M unprompted DisProt perplexity increases from 6.72 to 8.48 and prompted DisProt perplexity increases from 5.28 to 7.34. We additionally update the other DisProt evaluations so they are only done for these 750, finding no major changes: all biophysical/sequence metrics maintain the same trends as previously. We thank the Reviewer for bringing up these points.
We have additionally scanned the generated sequences for experimentally characterized functional sequence motifs, and we find that a high percentage of sequences generated after RL-SAE contain motifs that are expected within IDRs which associate to the target compartments. For these, we calculate motif occurrence rates for on- and off-target compartments, and we find that RL-SAE enriches the biologically expected motifs sequences for most compartments, with good specificity. We have updated Appendix P with this information.
To test whether the top-30 SAE features can help distinguish IDRs of target compartments from those of other compartments, we train several linear probes and small MLPs to perform target compartment classification of the ProtGPS IDRs. We test 3 input variants:
(1) only amino acid composition and sequence length (as a 21 element vector of 20 amino acid frequencies and log(sequence length))
(2) only top-30 SAE feature activation vectors (as a 30 element vector of maximum feature activation magnitudes)
(3) amino acid composition, sequence length, and SAE feature activation vectors all together (as a 51 element vector)
We train a linear probe and small MLP for each case, using a 70/30 data split per compartment (2171 IDRs total for SAE feature enrichment and probe training, and 982 held out), and we ensure that there are no shared parent proteins between the two splits. The table reports held-out AUROC using length and amino acid composition alone or additionally using top-30 SAE features (MLP hidden dimension = 32):
The results demonstrate that SAE feature activations do provide modest improvements beyond using only length + composition, increasing AUROC values on average across all 6 compartments by +0.03 for the linear probe and +0.035 for the MLP.