- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how frozen cross-modal contrastive models for molecules and proteins should be used (read out) for downstream prediction. Two representations are available in such models, the pooled encoder embedding and the projected embedding in the shared space, and it is observed that neither is uniformly better and that concatenating them gives limited gains. The authors propose to read out the frozen encoder tokens with a cross-attention whose query is derived from the projected embedding (Allosteric Readout), and to combine the result with the projected embedding through an input-conditioned gate (Homeostatic Fusion). The authors report the best average score in every task group on four frozen backbones (TRIDENT, S-PLM, ProteinCLIP and MoleculeSTM), including improvements over linear probes of larger biomolecular foundation models.
Critical strengths
S1. Application of attentive probing to frozen cross-modal biomolecular models
Attentive probing is the evaluation protocol in which a learned cross-attention aggregates the token embeddings of a frozen encoder, and it has been shown to outperform linear probing on frozen vision encoders (e.g. V-JEPA, AIM and the systematic study of Psomas et al., ICLR 2026). In this paper, it is applied to this domain, which helps practitioners increase the performance of frozen biomolecular models.
S2. Large and consistent empirical gains
Consistent gains over the reported baselines are obtained with ProbRead, although the evaluation protocol is substantially incomplete (see the weaknesses below).
Critical weaknesses
W1. Limited novelty and placing relative to the literature
The main contribution of the paper is the utilization of the projected embedding as the query of a multi-head cross-attention over the frozen encoder tokens (line 72, Section 3.2.1). This is attentive probing, which is by now a standard protocol for evaluating frozen encoders. It was proposed for masked image modelling [1] and is the default frozen evaluation of V-JEPA [2], AIM [3] and CAPI [4], and a systematic study at ICLR 2026 [5] shows that attentive probes consistently outperform linear probing. In the biomolecular domain, learned attention pooling over frozen protein language model embeddings [6] and learned readouts for molecular GNNs [7] report gains of the same kind. None of this work is discussed in Section 2, and no attentive probe is included in the experiments, although Appendix A cites Set Transformer, Perceiver and NV-Embed.
W2. Absence of proper baselining and misleading gains
Since the authors do not discuss the related literature on attentive probing (W1), they go to the natural next step of not baselining against this family of methods and thus reporting gains over "easier" baselines.
In terms of their actual benchmarking, this is still questionable. ProbRead is trained with AdamW, a cosine schedule and validation-based early stopping, whereas the TRIDENT linear probes use scikit-learn defaults, namely logistic regression with C = 1.0 and ridge regression with α = 1.0, with no tuning of the regularisation (Appendix C.3). Based on this, the claim in Section 4.1 that ProbRead and all baselines follow the same linear-probing protocol is therefore, in my current understanding, not accurate. The effect of this mismatch is visible in Table 4, where the Enc and Proj regressors perform close to a constant predictor. The table below is compiled from Table 4 of the paper, with the standard deviation of the labels added for reference. These standard deviations are taken from [8, Table 2] and are computed over the full MoleculeNet datasets, so they are only indicative for a scaffold test split.
| Dataset (units) | Label s.d. | Enc | Proj | MoLFormer row | ProbRead |
|---|---|---|---|---|---|
| FreeSolv (kcal/mol) | ≈ 3.85 | 4.176 | 3.968 | 3.308 | 2.198 |
| Lipophilicity (logD) | ≈ 1.20 | 1.041 | 1.030 | 0.933 | 0.792 |
| ESOL (log mol/L) | ≈ 2.10 | 1.657 | 1.660 | 1.637 | 1.355 |
This table, along with other information from the paper, indicates no proper implementation of the presented baselines, since:
- On FreeSolv, the Enc and Proj errors are larger than the label standard deviation, which means they are no better than predicting the mean. The linear probe on MoLFormer in the same table, which has the same architecture as the TRIDENT encoder, is clearly better.
- The Enc probe on MoleculeSTM reaches 48.7 AUC on ClinTox (Table 6), which is below chance.
- In the cases where the linear probe is strong, the advantage disappears, with ProbRead below Enc on HIV (77.9 against 79.9), BACE (79.6 against 80.2) and Carcinogens (94.4 against 97.5).
Part of the reported improvement, for example the reduction of average RMSE from 1.882 to 1.212, therefore includes weak baselines.
The comparison with other foundation models has the same problem. ProbRead, a trained attentive readout, is compared with linear probes on mean-pooled embeddings of the other models. On ProteinCLIP, the encoder is the original frozen ESM-2 150M [9], so the CATH result in Figure 5 (73.64 against 70.53 for ESM-2 650M) compares two readouts of ESM-2, not two models. The claim of outperforming other foundation models only holds if the same readout is applied to their tokens.
W3. Questionable practical benefit
The frozen setting is motivated in Section 1 by the cost of fine-tuning, yet fine-tuning is never evaluated. Parameter-efficient fine-tuning, including adapters, bias or normalisation tuning and low-rank adaptation [10], is not discussed either, even though attentive probing itself has been described as a lightweight additive form of it [5]. This baselining is significant for this paper since
- LoRA on ESM-2 is competitive with full fine-tuning at a fraction of the memory [11] and outperforms frozen embeddings with a prediction head on most protein tasks [12].
- The backbones in this paper are small, with 47M parameters for MoLFormer-XL and 150M for ESM-2 in ProteinCLIP, and S-PLM is itself an adapter-augmented ESM-2.
- ProbRead trains 4.1M parameters on TRIDENT, which is 8.7% of the encoder (Table 8) and more than a standard LoRA configuration would train on a model of this size.
At the other end, the method is not compared in cost with the simple alternative of training two linear probes and keeping the better one.
ProbRead requires storing the full token sequence of every example, which for S-PLM is one 1280d vector per residue instead of one per protein, and training a cross-attention module for every task. Without fine-tuning or LoRA as an upper bound and a table of trainable parameters, memory, storage and training time, it is not possible to judge whether ProbRead offers a better accuracy-cost trade-off than either alternative.
W4. Weak motivation
The method is motivated by Figure 1 where the better of Enc and Proj changes across tasks. Since the downstream task labels are not known during pretraining, a probe needs to be trained for the downstream task anyway. Why not just solve the issue by training one linear probe on the concatenated embeddings, or by training two linear probes and keeping the one that is better on the validation set? Both options cost one or two linear fits on cached vectors, so there is no need for a readout that works universally across tasks.
The projection bottleneck is also largely known. The loss of information through the projection head is documented in works that are cited in the paper (references [12] to [14] and [51] of the paper), and the ProteinCLIP paper already reports that its projected embeddings improve PPI prediction over raw ESM-2 embeddings [9]. Moreover, Figure 4 compares the raw effective rank of the encoder and the projector, which is not a fair comparison as seen in the table below which is compiled from Tables 8 and 9 of the paper
| Platform | Dataset | Encoder (erank / dim) | Projector (erank / dim) |
|---|---|---|---|
| TRIDENT | CYP2C19 | 436.7 / 768 = 57% | 220.2 / 512 = 43% |
| TRIDENT | Tox21 | 386.3 / 768 = 50% | 206.7 / 512 = 40% |
| TRIDENT | ToxCast | 397.3 / 768 = 52% | 200.3 / 512 = 39% |
| S-PLM | EC | 602.7 / 1280 = 47% | 48.1 / 256 = 19% |
| S-PLM | GO | 680.1 / 1280 = 53% | 47.8 / 256 = 19% |
| ProteinCLIP | PPI | 315.9 / 640 = 49% | 87.5 / 128 = 68% |
On ProteinCLIP, the projector uses a larger fraction of its dimensions than the encoder (68% against 49%), so the drop in raw effective rank from 316 to 88 is explained entirely by the projector having 128 dimensions instead of 640, and no compression beyond that occurs. The claim in Section 4.2 that the projection bottleneck manifests consistently across architectures therefore holds for raw effective rank but not for the fraction of dimensions used. Contribution (i) should be presented as a confirmation of known effects in the biomolecular setting, with the dimension-normalised effective rank reported next to Figure 4.
W5. Unclear presentation
The paper is difficult to follow because the method is explained through physical analogies rather than through what it computes. Allosteric Readout (AR) is introduced through allosteric regulation, in which a regulator binds at a site distinct from the active site and modulates activity there without being the substrate. Terms such as active site, substrate and modulation of activity are never mapped onto the readout, so an ML reader must decode the analogy before understanding the method.
References
[1] X. Chen et al. Context Autoencoder for Self-Supervised Representation Learning. International Journal of Computer Vision 132:208-223, 2024. arXiv:2202.03026.
[2] A. Bardes et al. Revisiting Feature Prediction for Learning Visual Representations from Video. Transactions on Machine Learning Research, 2024. arXiv:2404.08471.
[3] A. El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models. ICML 2024. arXiv:2401.08541.
[4] T. Darcet et al. Cluster and Predict Latent Patches for Improved Masked Image Modeling. Transactions on Machine Learning Research, 2025. arXiv:2502.08769.
[5] B. Psomas et al. Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency. ICLR 2026. arXiv:2506.10178.
[6] H. Stärk, C. Dallago, M. Heinzinger, B. Rost. Light attention predicts protein location from the language of life. Bioinformatics Advances 1(1):vbab035, 2021.
[7] D. Buterez, J. P. Janet, S. J. Kiddle, D. Oglic, P. Liò. Graph Neural Networks with Adaptive Readouts. NeurIPS 2022. arXiv:2211.04952.
[8] Y. Zhang, J. Menke, J. He et al. Similarity-based pairing improves efficiency of siamese neural networks for regression tasks and uncertainty quantification. Journal of Cheminformatics 15:75, 2023.
[9] K. E. Wu, H. Chang, J. Zou. ProteinCLIP: enhancing protein language models with natural language. bioRxiv 2024.05.14.594226, 2024.
[10] E. J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685.
[11] S. Sledzieski et al. Democratizing protein language models with parameter-efficient fine-tuning. Proceedings of the National Academy of Sciences 121(26):e2405840121, 2024.
[12] R. Schmirler, M. Heinzinger, B. Rost. Fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications 15:7407, 2024.
Questions for authors
Questions For Authors
I understand that the critical weaknesses identified in my review are difficult to address within a rebuttal period, and that, if addressed, the resulting paper would differ significantly from the initial submission. However, I write my questions and suggestions here in an attempt to help the authors improve their submission.
-
Positioning within the attentive probing literature (W1). It would strengthen the paper to discuss attentive probing [1, 2, 3] and learned readouts over frozen biomolecular encoders [4, 5] in Section 2, and to state clearly that the contribution is the use of the projected embedding as the query of an attentive probe.
-
Baselining against attentive probing (W1, W2). ProbRead should be compared with a learnable-query attentive probe of matched capacity, in the standard multi-head form of [2] and in the efficient form of [3], trained with exactly the ProbRead optimiser, schedule and early stopping. In the same comparison, the linear probes should be tuned on the validation split and seed standard deviations should be reported. The same attentive probe should also be applied to the tokens of the external foundation models, so that the comparison in Figure 5 is between models rather than between readouts.
-
Baselining against fine-tuning (W3). Since the frozen setting is motivated by the cost of fine-tuning, full fine-tuning should be reported as an upper bound, at least on the two smallest backbones, MoLFormer-XL (47M) and ESM-2 150M.
-
Baselining against parameter-efficient fine-tuning (W3). It would strengthen the paper to compare with low-rank adaptation [6] at a trainable parameter budget comparable to ProbRead, following the protocols of [7, 8] for the protein tasks, and with adapter tuning, which S-PLM already uses on top of ESM-2.
-
Computational and memory comparison (W3, W4). A table of trainable parameters, peak memory, storage of cached features and wall-clock training time should place ProbRead against a linear probe on the concatenated embeddings, two linear probes with validation selection, an attentive probe, LoRA and full fine-tuning. This would show whether the gains justify caching the full token sequence of every example.
-
Restructuring the effective-rank argument (W4). The effective rank in Figure 4 should be reported normalised by the dimension of each representation, and the claim in Section 4.2 that the projection bottleneck manifests consistently across architectures should be restricted to the platforms where the normalised effective rank actually drops, namely TRIDENT and, most clearly, S-PLM. Contribution (i) would then be presented as a confirmation of known projection-head effects in the biomolecular setting.
References
[1] X. Chen et al. Context Autoencoder for Self-Supervised Representation Learning. International Journal of Computer Vision 132:208-223, 2024. arXiv:2202.03026.
[2] A. Bardes et al. Revisiting Feature Prediction for Learning Visual Representations from Video. Transactions on Machine Learning Research, 2024. arXiv:2404.08471.
[3] B. Psomas et al. Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency. ICLR 2026. arXiv:2506.10178.
[4] H. Stärk, C. Dallago, M. Heinzinger, B. Rost. Light attention predicts protein location from the language of life. Bioinformatics Advances 1(1):vbab035, 2021.
[5] D. Buterez, J. P. Janet, S. J. Kiddle, D. Oglic, P. Liò. Graph Neural Networks with Adaptive Readouts. NeurIPS 2022. arXiv:2211.04952.
[6] E. J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685.
[7] S. Sledzieski et al. Democratizing protein language models with parameter-efficient fine-tuning. Proceedings of the National Academy of Sciences 121(26):e2405840121, 2024.
[8] R. Schmirler, M. Heinzinger, B. Rost. Fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications 15:7407, 2024.
Follow-up: Request for Feedback on Our Rebuttal
Dear Reviewer mYZQ,
Thank you again for your thoughtful review. As the author–reviewer discussion period draws to a close, we would greatly appreciate it if you could take a look at our responses and additional experiments and share your updated assessment. We would be happy to address any remaining concerns.
Thank you for your time and consideration.