acceptodds
Under review as a conference paper at ICLR 2027

ProbRead: Allosteric Readout and Homeostatic Fusion for Frozen Cross-Modal Biomolecular Models

Abstract

Cross-modal biomolecular models for molecules and proteins use an encoder for each modality and a projector that maps the pooled encoder output into a shared space. For downstream tasks, these models are typically frozen and evaluated by linear probing on a single representation: the pooled encoder embedding or the projected embedding. We observe that the better representation changes across tasks. Moreover, fixed pooling applies the same task-independent rule to every input, and concatenating the two pooled embeddings yields limited gains. We propose ProbRead (Probe Readout), a lightweight readout with two components: (i) Allosteric Readout uses the projected embedding only to guide token selection over the encoder tokens, replacing fixed pooling. (ii) Homeostatic Fusion assigns complementary, input-conditioned weights to the aggregated token representation and the projected embedding in each feature dimension. Across three platforms and five groups of biomolecular tasks, ProbRead achieves the best average performance, outperforming encoder-only and projector-only probing as well as linear probing of other biomolecular foundation models. ProbRead 's output also recovers the effective rank lost by projection. These results demonstrate that adaptively combining the two representations is more effective than fixing either one.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
cross-modal contrastive learning, projection head, information bottleneck, linear probing, token-level readout, effective rank, representation learning, molecular property prediction, protein function prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5x8pOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how frozen cross-modal contrastive models for molecules and proteins should be used (read out) for downstream prediction. Two representations are available in such models, the pooled encoder embedding and the projected embedding in the shared space, and it is observed that neither is uniformly better and that concatenating them gives limited gains. The authors propose to read out the frozen encoder tokens with a cross-attention whose query is derived from the projected embedding (Allosteric Readout), and to combine the result with the projected embedding through an input-conditioned gate (Homeostatic Fusion). The authors report the best average score in every task group on four frozen backbones (TRIDENT, S-PLM, ProteinCLIP and MoleculeSTM), including improvements over linear probes of larger biomolecular foundation models.

Critical strengths

S1. Application of attentive probing to frozen cross-modal biomolecular models

Attentive probing is the evaluation protocol in which a learned cross-attention aggregates the token embeddings of a frozen encoder, and it has been shown to outperform linear probing on frozen vision encoders (e.g. V-JEPA, AIM and the systematic study of Psomas et al., ICLR 2026). In this paper, it is applied to this domain, which helps practitioners increase the performance of frozen biomolecular models.

S2. Large and consistent empirical gains

Consistent gains over the reported baselines are obtained with ProbRead, although the evaluation protocol is substantially incomplete (see the weaknesses below).

Critical weaknesses

W1. Limited novelty and placing relative to the literature

The main contribution of the paper is the utilization of the projected embedding as the query of a multi-head cross-attention over the frozen encoder tokens (line 72, Section 3.2.1). This is attentive probing, which is by now a standard protocol for evaluating frozen encoders. It was proposed for masked image modelling [1] and is the default frozen evaluation of V-JEPA [2], AIM [3] and CAPI [4], and a systematic study at ICLR 2026 [5] shows that attentive probes consistently outperform linear probing. In the biomolecular domain, learned attention pooling over frozen protein language model embeddings [6] and learned readouts for molecular GNNs [7] report gains of the same kind. None of this work is discussed in Section 2, and no attentive probe is included in the experiments, although Appendix A cites Set Transformer, Perceiver and NV-Embed.

W2. Absence of proper baselining and misleading gains

Since the authors do not discuss the related literature on attentive probing (W1), they go to the natural next step of not baselining against this family of methods and thus reporting gains over "easier" baselines.

In terms of their actual benchmarking, this is still questionable. ProbRead is trained with AdamW, a cosine schedule and validation-based early stopping, whereas the TRIDENT linear probes use scikit-learn defaults, namely logistic regression with C = 1.0 and ridge regression with α = 1.0, with no tuning of the regularisation (Appendix C.3). Based on this, the claim in Section 4.1 that ProbRead and all baselines follow the same linear-probing protocol is therefore, in my current understanding, not accurate. The effect of this mismatch is visible in Table 4, where the Enc and Proj regressors perform close to a constant predictor. The table below is compiled from Table 4 of the paper, with the standard deviation of the labels added for reference. These standard deviations are taken from [8, Table 2] and are computed over the full MoleculeNet datasets, so they are only indicative for a scaffold test split.

Dataset (units)Label s.d.EncProjMoLFormer rowProbRead
FreeSolv (kcal/mol)≈ 3.854.1763.9683.3082.198
Lipophilicity (logD)≈ 1.201.0411.0300.9330.792
ESOL (log mol/L)≈ 2.101.6571.6601.6371.355

This table, along with other information from the paper, indicates no proper implementation of the presented baselines, since:

  • On FreeSolv, the Enc and Proj errors are larger than the label standard deviation, which means they are no better than predicting the mean. The linear probe on MoLFormer in the same table, which has the same architecture as the TRIDENT encoder, is clearly better.
  • The Enc probe on MoleculeSTM reaches 48.7 AUC on ClinTox (Table 6), which is below chance.
  • In the cases where the linear probe is strong, the advantage disappears, with ProbRead below Enc on HIV (77.9 against 79.9), BACE (79.6 against 80.2) and Carcinogens (94.4 against 97.5).

Part of the reported improvement, for example the reduction of average RMSE from 1.882 to 1.212, therefore includes weak baselines.

The comparison with other foundation models has the same problem. ProbRead, a trained attentive readout, is compared with linear probes on mean-pooled embeddings of the other models. On ProteinCLIP, the encoder is the original frozen ESM-2 150M [9], so the CATH result in Figure 5 (73.64 against 70.53 for ESM-2 650M) compares two readouts of ESM-2, not two models. The claim of outperforming other foundation models only holds if the same readout is applied to their tokens.

W3. Questionable practical benefit

The frozen setting is motivated in Section 1 by the cost of fine-tuning, yet fine-tuning is never evaluated. Parameter-efficient fine-tuning, including adapters, bias or normalisation tuning and low-rank adaptation [10], is not discussed either, even though attentive probing itself has been described as a lightweight additive form of it [5]. This baselining is significant for this paper since

  • LoRA on ESM-2 is competitive with full fine-tuning at a fraction of the memory [11] and outperforms frozen embeddings with a prediction head on most protein tasks [12].
  • The backbones in this paper are small, with 47M parameters for MoLFormer-XL and 150M for ESM-2 in ProteinCLIP, and S-PLM is itself an adapter-augmented ESM-2.
  • ProbRead trains 4.1M parameters on TRIDENT, which is 8.7% of the encoder (Table 8) and more than a standard LoRA configuration would train on a model of this size.

At the other end, the method is not compared in cost with the simple alternative of training two linear probes and keeping the better one.

ProbRead requires storing the full token sequence of every example, which for S-PLM is one 1280d vector per residue instead of one per protein, and training a cross-attention module for every task. Without fine-tuning or LoRA as an upper bound and a table of trainable parameters, memory, storage and training time, it is not possible to judge whether ProbRead offers a better accuracy-cost trade-off than either alternative.

W4. Weak motivation

The method is motivated by Figure 1 where the better of Enc and Proj changes across tasks. Since the downstream task labels are not known during pretraining, a probe needs to be trained for the downstream task anyway. Why not just solve the issue by training one linear probe on the concatenated embeddings, or by training two linear probes and keeping the one that is better on the validation set? Both options cost one or two linear fits on cached vectors, so there is no need for a readout that works universally across tasks.

The projection bottleneck is also largely known. The loss of information through the projection head is documented in works that are cited in the paper (references [12] to [14] and [51] of the paper), and the ProteinCLIP paper already reports that its projected embeddings improve PPI prediction over raw ESM-2 embeddings [9]. Moreover, Figure 4 compares the raw effective rank of the encoder and the projector, which is not a fair comparison as seen in the table below which is compiled from Tables 8 and 9 of the paper

PlatformDatasetEncoder (erank / dim)Projector (erank / dim)
TRIDENTCYP2C19436.7 / 768 = 57%220.2 / 512 = 43%
TRIDENTTox21386.3 / 768 = 50%206.7 / 512 = 40%
TRIDENTToxCast397.3 / 768 = 52%200.3 / 512 = 39%
S-PLMEC602.7 / 1280 = 47%48.1 / 256 = 19%
S-PLMGO680.1 / 1280 = 53%47.8 / 256 = 19%
ProteinCLIPPPI315.9 / 640 = 49%87.5 / 128 = 68%

On ProteinCLIP, the projector uses a larger fraction of its dimensions than the encoder (68% against 49%), so the drop in raw effective rank from 316 to 88 is explained entirely by the projector having 128 dimensions instead of 640, and no compression beyond that occurs. The claim in Section 4.2 that the projection bottleneck manifests consistently across architectures therefore holds for raw effective rank but not for the fraction of dimensions used. Contribution (i) should be presented as a confirmation of known effects in the biomolecular setting, with the dimension-normalised effective rank reported next to Figure 4.

W5. Unclear presentation

The paper is difficult to follow because the method is explained through physical analogies rather than through what it computes. Allosteric Readout (AR) is introduced through allosteric regulation, in which a regulator binds at a site distinct from the active site and modulates activity there without being the substrate. Terms such as active site, substrate and modulation of activity are never mapped onto the readout, so an ML reader must decode the analogy before understanding the method.

References

[1] X. Chen et al. Context Autoencoder for Self-Supervised Representation Learning. International Journal of Computer Vision 132:208-223, 2024. arXiv:2202.03026.

[2] A. Bardes et al. Revisiting Feature Prediction for Learning Visual Representations from Video. Transactions on Machine Learning Research, 2024. arXiv:2404.08471.

[3] A. El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models. ICML 2024. arXiv:2401.08541.

[4] T. Darcet et al. Cluster and Predict Latent Patches for Improved Masked Image Modeling. Transactions on Machine Learning Research, 2025. arXiv:2502.08769.

[5] B. Psomas et al. Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency. ICLR 2026. arXiv:2506.10178.

[6] H. Stärk, C. Dallago, M. Heinzinger, B. Rost. Light attention predicts protein location from the language of life. Bioinformatics Advances 1(1):vbab035, 2021.

[7] D. Buterez, J. P. Janet, S. J. Kiddle, D. Oglic, P. Liò. Graph Neural Networks with Adaptive Readouts. NeurIPS 2022. arXiv:2211.04952.

[8] Y. Zhang, J. Menke, J. He et al. Similarity-based pairing improves efficiency of siamese neural networks for regression tasks and uncertainty quantification. Journal of Cheminformatics 15:75, 2023.

[9] K. E. Wu, H. Chang, J. Zou. ProteinCLIP: enhancing protein language models with natural language. bioRxiv 2024.05.14.594226, 2024.

[10] E. J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685.

[11] S. Sledzieski et al. Democratizing protein language models with parameter-efficient fine-tuning. Proceedings of the National Academy of Sciences 121(26):e2405840121, 2024.

[12] R. Schmirler, M. Heinzinger, B. Rost. Fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications 15:7407, 2024.

Questions for authors

Questions For Authors

I understand that the critical weaknesses identified in my review are difficult to address within a rebuttal period, and that, if addressed, the resulting paper would differ significantly from the initial submission. However, I write my questions and suggestions here in an attempt to help the authors improve their submission.

  1. Positioning within the attentive probing literature (W1). It would strengthen the paper to discuss attentive probing [1, 2, 3] and learned readouts over frozen biomolecular encoders [4, 5] in Section 2, and to state clearly that the contribution is the use of the projected embedding as the query of an attentive probe.

  2. Baselining against attentive probing (W1, W2). ProbRead should be compared with a learnable-query attentive probe of matched capacity, in the standard multi-head form of [2] and in the efficient form of [3], trained with exactly the ProbRead optimiser, schedule and early stopping. In the same comparison, the linear probes should be tuned on the validation split and seed standard deviations should be reported. The same attentive probe should also be applied to the tokens of the external foundation models, so that the comparison in Figure 5 is between models rather than between readouts.

  3. Baselining against fine-tuning (W3). Since the frozen setting is motivated by the cost of fine-tuning, full fine-tuning should be reported as an upper bound, at least on the two smallest backbones, MoLFormer-XL (47M) and ESM-2 150M.

  4. Baselining against parameter-efficient fine-tuning (W3). It would strengthen the paper to compare with low-rank adaptation [6] at a trainable parameter budget comparable to ProbRead, following the protocols of [7, 8] for the protein tasks, and with adapter tuning, which S-PLM already uses on top of ESM-2.

  5. Computational and memory comparison (W3, W4). A table of trainable parameters, peak memory, storage of cached features and wall-clock training time should place ProbRead against a linear probe on the concatenated embeddings, two linear probes with validation selection, an attentive probe, LoRA and full fine-tuning. This would show whether the gains justify caching the full token sequence of every example.

  6. Restructuring the effective-rank argument (W4). The effective rank in Figure 4 should be reported normalised by the dimension of each representation, and the claim in Section 4.2 that the projection bottleneck manifests consistently across architectures should be restricted to the platforms where the normalised effective rank actually drops, namely TRIDENT and, most clearly, S-PLM. Contribution (i) would then be presented as a confirmation of known projection-head effects in the biomolecular setting.

References

[1] X. Chen et al. Context Autoencoder for Self-Supervised Representation Learning. International Journal of Computer Vision 132:208-223, 2024. arXiv:2202.03026.

[2] A. Bardes et al. Revisiting Feature Prediction for Learning Visual Representations from Video. Transactions on Machine Learning Research, 2024. arXiv:2404.08471.

[3] B. Psomas et al. Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency. ICLR 2026. arXiv:2506.10178.

[4] H. Stärk, C. Dallago, M. Heinzinger, B. Rost. Light attention predicts protein location from the language of life. Bioinformatics Advances 1(1):vbab035, 2021.

[5] D. Buterez, J. P. Janet, S. J. Kiddle, D. Oglic, P. Liò. Graph Neural Networks with Adaptive Readouts. NeurIPS 2022. arXiv:2211.04952.

[6] E. J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685.

[7] S. Sledzieski et al. Democratizing protein language models with parameter-efficient fine-tuning. Proceedings of the National Academy of Sciences 121(26):e2405840121, 2024.

[8] R. Schmirler, M. Heinzinger, B. Rost. Fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications 15:7407, 2024.

Reviewer_mYZQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProbRead, a lightweight readout for frozen cross-modal biomolecular models. Rather than linearly probing either the pooled encoder or projector vector, it uses the projected vector to select encoder tokens and fuses them with the projection via input- and dimension-specific weights. Across three model platforms and five task categories, it achieves the best average performance and restores the effective rank lost through projection.

Critical strengths
  • The paper identifies a previously unaddressed trade-off: prior work evaluates either the projected vector (CLIP, TRIDENT, ProteinCLIP) or, following SimCLR, discards the projection head in favor of the encoder vector (MoleculeSTM, ProtST, ProTrek), yet the authors show through three lines of evidence (task-dependent reversals within the same model, marked dimensional collapse after projection in effective-rank analysis, and limited, unstable gains from simply concatenating the two vectors) that neither choice is universally optimal.

  • With all backbones frozen, ProbRead achieves the best average performance in every one of five biomolecular task categories across three cross-modal platforms, outperforming encoder-only and projector-only probes as well as linear probes of other pretrained models, while retaining 81–123% of the encoder’s effective rank, indicating no dimensional collapse.

Critical weaknesses

The paper includes no component ablations, so it remains unclear whether the projected vector z is a better query than a learnable query (the standard approach in CLIP, Set Transformer, and Perceiver) or the pooled encoder vector, and how AR alone, HF alone, or HF replaced by concatenation or a scalar gate would perform.

Questions for authors
  • What are the results when z is replaced with a learnable query, or with the pooled encoder vector as the query?
1 comment
AuthorsOpenReview ↗

Follow-up: Request for Feedback on Our Rebuttal

Dear Reviewer mYZQ,

Thank you again for your thoughtful review. As the author–reviewer discussion period draws to a close, we would greatly appreciate it if you could take a look at our responses and additional experiments and share your updated assessment. We would be happy to address any remaining concerns.

Thank you for your time and consideration.

Reviewer_8ZJkOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method called ProbRead (Probe Readout) that is meant to be a way to bring together pretrained AI models for molecules and for proteins. It has two readout components that are inspired by biology itself: Allosteric Readout uses the projected embedding only to guide token selection over the encoder tokens, replacing fixed pooling, whereas Homeostatic Fusion assigns complementary, input-conditioned weights to the aggregated token representation and the projected embedding in each feature dimension. Through extensive experiments, this is demonstrated to be effective in several downstream tasks. The core idea is not to freeze everything. Overall, the results demonstrate adaptively combining representations is more effective than fixing either one.

Critical strengths

The core of small-molecule drug discovery is molecule-protein interaction, so no question the problem addressed is important.

The study is well-motivated theoretically (interestingly bio-inspired) and the developed technique is demonstrated to be quite efficacious across a wide variety of experiments.

Critical weaknesses

There are other seemingly stat-of-the-art molecule models, such as "Foundation Models for Discovery and Exploration in Chemical Space" by Wadell et al. Would be curious to see how that plays into the presently-proposed architecture.

No other specific weaknesses noted.

Questions for authors

How is the proposed approach related to other recent approaches to multimodal fusion, e.g. the "DeepInsert" approach of Choraria et al (EACL 2026) and its theoretical follow-on work, "Skip It?" (ICML 2026)? Would be good to get this clarification, especially the relation of the bioinspired approach and the information-theoretic approach.

Comments on the paper

4 comments

I thank the authors for the further experiments, responses and continual engagement with my comments.

1. Result-wise. Against the capacity-matched probe, ProbRead separates clearly on Tox21, BACE, FreeSolv, CYP2C19 and both S-PLM function tasks, and is clearly behind only on Lipophilicity, with the remaining comparisons falling within the reported seed variance. On the regression group it holds the lower mean on three of the four tasks, so I accept that the task-level ordering there favours it even though the raw average is carried by FreeSolv. At group level the picture is narrower, with MoleculeNet classification and the S-PLM tasks separating while sequence-grouped CATH, TRIDENT TDC and MIST TDC remain within variance. Three further groups carry no such control, namely MoleculeSTM, the GO branches other than MF, and ProteinCLIP PPI, the last of which is the task used in Figure 1 to motivate the work. There is a signal that the method may outperform the alternatives, although establishing it is not something that can be completed within a rebuttal period.

2. Method-wise. The level of methodological contribution remains arguable. On the evidence now available, the contribution reads to me as the transfer of attentive probing to frozen biomolecular models with the projector used as the query generator. Since the attention and the gated fusion are both standard, what is new is the source of the query, which I would describe as a choice of conditioning signal. It should be noted that the methodological and the empirical assessments are not separable here. A small change of mechanism that yields a substantial and consistent empirical gain would not be a minor contribution, whereas the transfer of a known method without a clear benefit is a limited one.

3. Efficiency-wise. I thank the authors for the clarification. The comparable memory and storage follow from the capacity matching, and at that matched budget the performance is also comparable on four of the six groups. A sweep over trainable-parameter budgets would settle the question, since ProbRead should hold its performance at budgets where MHAP degrades if the conditioning is the source of the benefit, although I recognise that there is no time for this within the discussion period.

Overall. A rebuttal period usually rests on the provision of further ablations over the methodological components and on the clarification of the claims being made. This discussion has instead rested on establishing whether the method actually performs better on the main benchmarks, the submitted evaluation having lacked tuned baselines and any attentive-probe control. What the paper requires is comparisons carrying full fine-tuning as the oracle, low-rank adaptation as the standard parameter-efficient reference, multi-head attention pooling as the standard attentive probe, and plausibly an input-conditioned readout such as Light Attention, reported across every task group of the paper rather than across a different subset of tasks for each baseline. As matters stand these comparisons were begun during the discussion period and each covers a different selection of tasks. That indicates a submission which is not yet at the stage where a rebuttal can settle it, since what remains open is the magnitude of the reported improvements and their coverage across the paper's benchmarks, rather than how they are to be interpreted. The authors have nonetheless done a great deal to address several of my comments, and I therefore retain my raise from clear rejection to weak rejection.

AuthorsOpenReview ↗

Response to Reviewer 5x8p – Part 11

We thank the reviewer for the continued discussion and for recognising the task-level evidence. Below we provide additional comparisons with multi-head attention pooling (MHAP) and address the methodological contribution, parameter-budget evaluation and baseline coverage.


Q18. Result-wise. Against the capacity-matched probe, ProbRead separates clearly on Tox21, BACE, FreeSolv, CYP2C19 and both S-PLM function tasks, and is clearly behind only on Lipophilicity, with the remaining comparisons falling within the reported seed variance. On the regression group it holds the lower mean on three of the four tasks, so I accept that the task-level ordering there favours it even though the raw average is carried by FreeSolv. At group level the picture is narrower, with MoleculeNet classification and the S-PLM tasks separating while sequence-grouped CATH, TRIDENT TDC and MIST TDC remain within variance. Three further groups carry no such control, namely MoleculeSTM, the GO branches other than MF, and ProteinCLIP PPI, the last of which is the task used in Figure 1 to motivate the work. There is a signal that the method may outperform the alternatives, although establishing it is not something that can be completed within a rebuttal period.

A18. We add capacity-matched MHAP comparisons on MoleculeSTM and ProteinCLIP PPI, and extend the S-PLM evaluation to the GO biological process (GO-BP) and cellular component (GO-CC) branches.

(i) MoleculeSTM, molecular classification, AUROC (%) ↑

MethodBACEBBBPClinToxHIVMUVSIDERTox21Avg
MHAP68.21 ± 0.9466.34 ± 2.4962.92 ± 1.4873.53 ± 2.0067.42 ± 0.7963.55 ± 0.1072.52 ± 0.6567.78 ± 0.20
ProbRead76.18 ± 1.4867.06 ± 1.2975.38 ± 3.3273.93 ± 0.8769.88 ± 1.9162.24 ± 0.8474.28 ± 0.7171.28 ± 0.67

ProbRead achieves a higher group average and better mean performance on most tasks, while MHAP performs better on SIDER. The overall advantage extends the evidence to a graph-based molecular encoder and directly addresses the missing MoleculeSTM control.

(ii) ProteinCLIP, PPI prediction, AUROC (%) ↑

MethodPPI
MHAP67.67 ± 0.26
ProbRead68.36 ± 0.41

ProbRead also has the higher mean on PPI, with a modest margin. This comparison directly tests the readout choice on the task used to motivate Figure 1.

(iii) S-PLM, additional GO branches, Fmax ↑

MethodGO-BPGO-CC
MHAP0.3655 ± 0.00690.3791 ± 0.0441
ProbRead0.3852 ± 0.00010.3924 ± 0.0024

ProbRead has the higher mean on both additional GO branches. The margin is clearer on BP, whereas the CC comparison remains within MHAP's larger seed variability. Together with the EC and GO-MF results in A14, these experiments broaden the evidence for projection-guided retrieval and fusion across protein-function tasks.

Taken together, the capacity-matched comparisons in A1, A14 and the additional experiments above show gains over MHAP on molecular classification and protein-function prediction, with the new MoleculeSTM comparison adding a clear group-level advantage on a graph-based encoder. ProbRead also achieves a better mean on most individual tasks, although the margin is less pronounced on some. The overall pattern supports ProbRead's advantage over a standard MHAP readout.

AuthorsOpenReview ↗

Response to Reviewer 5x8p – Part 12

Q19. Method-wise. The level of methodological contribution remains arguable. On the evidence now available, the contribution reads to me as the transfer of attentive probing to frozen biomolecular models with the projector used as the query generator. Since the attention and the gated fusion are both standard, what is new is the source of the query, which I would describe as a choice of conditioning signal. It should be noted that the methodological and the empirical assessments are not separable here. A small change of mechanism that yields a substantial and consistent empirical gain would not be a minor contribution, whereas the transfer of a known method without a clear benefit is a limited one.

A19. ProbRead's contribution is a readout design that connects cross-modal alignment, local token retrieval and adaptive representation fusion. The projected embedding has two coordinated roles, which address the encoder–projector trade-off identified in the paper.

(i) Cross-modal alignment determines the retrieval condition. MHAP uses a learned query shared across samples and input-dependent token keys. ProbRead makes the query itself depend on each sample's pretrained cross-modal projection. Consequently, the compatibility function used to select token content varies with the aligned global representation. Light Attention instead generates token scores through local convolutions. ProbRead turns the knowledge encoded by cross-modal pretraining into a condition for retrieving downstream-relevant molecular or residue features.

(ii) The same aligned representation remains available as predictive content. Allosteric Readout (AR) uses the projection to select encoder content; Homeostatic Fusion (HF) then combines that content with a transformed projection through complementary, input-dependent weights for each feature dimension. This joint design retains access to both shared cross-modal information and modality-specific encoder detail. It addresses the task-dependent utility of encoder and projector representations through both information selection and information balancing.

Our query-source and fusion ablations support these two design choices. The capacity-matched comparisons in A1, A14 and A18 further show their practical value, with the clearest gains on MoleculeNet classification, S-PLM function prediction and MoleculeSTM. The methodological contribution is therefore the coordinated use of a pretrained alignment representation to guide retrieval and participate in prediction, supported by gains over a standard attentive probe across multiple biomolecular settings.


Q20. Efficiency-wise. I thank the authors for the clarification. The comparable memory and storage follow from the capacity matching, and at that matched budget the performance is also comparable on four of the six groups. A sweep over trainable-parameter budgets would settle the question, since ProbRead should hold its performance at budgets where MHAP degrades if the conditioning is the source of the benefit, although I recognise that there is no time for this within the discussion period.

A20. Our completed capacity-matched comparisons show gains over multi-head attention pooling (MHAP) on molecular classification and protein-function prediction, including the new MoleculeSTM comparison in A18. These results provide evidence that ProbRead's readout contributes beyond MHAP. Your proposed sweep over trainable-parameter budgets is an interesting way to test whether projection-conditioned retrieval helps ProbRead retain performance better than MHAP at lower parameter budgets. We could not complete this additional experiment within the time available for this rebuttal. We will add these additional experiments in the final version.

AuthorsOpenReview ↗

Response to Reviewer 5x8p – Part 13

Q21. What the paper requires is comparisons carrying full fine-tuning as the oracle, low-rank adaptation as the standard parameter-efficient reference, multi-head attention pooling as the standard attentive probe, and plausibly an input-conditioned readout such as Light Attention, reported across every task group of the paper rather than across a different subset of tasks for each baseline. As matters stand these comparisons were begun during the discussion period and each covers a different selection of tasks. That indicates a submission which is not yet at the stage where a rebuttal can settle it, since what remains open is the magnitude of the reported improvements and their coverage across the paper's benchmarks, rather than how they are to be interpreted. The authors have nonetheless done a great deal to address several of my comments, and I therefore retain my raise from clear rejection to weak rejection.

A21. We understand the request for consistent coverage across task groups. The capacity-matched multi-head attention pooling comparisons now cover all eight task groups: TRIDENT TDC, MoleculeNet and regression; S-PLM EC and GO-MF; ProteinCLIP PPI and sequence-grouped CATH; and MoleculeSTM. A18 provides the additional comparisons that complete this coverage.

The performance comparisons with full fine-tuning (FullFT) and low-rank adaptation (LoRA) remain partial. We previously reported their Caco-2 cost measurements alongside ProbRead, summarized below.

Caco-2, resource measurements averaged over three seeds

MethodTrainable parametersPeak GPU memory (GiB)Recorded wall-clock time
LoRA4,129,5370.6572117.42 s
FullFT44,375,8091.1151114.80 s
ProbRead4,136,4490.169212.22 s

We additionally extend the resource comparison to the larger S-PLM EC setting. Given the long fine-tuning runtime, this additional cost estimate uses the mean wall-clock time over the first three epochs of seed 0 to assess the computational cost of completing broader comparisons.

S-PLM EC, seed 0, timing averaged over the first three epochs

MethodTrainable parametersPeak GPU memory (GiB)Mean wall-clock time per epoch
LoRA9,813,0183.3575.77 min
FullFT702,661,65813.4869.50 min
ProbRead9,875,7380.6636.98 s

All runs use A100-80GB GPUs. Caco-2 reports complete training wall-clock time; S-PLM EC timing is averaged over the first three epochs, with peak memory from the seed-0 resource measurement. Frozen-readout time covers training on cached features and excludes initial backbone encoding and shared feature loading.

These measurements demonstrate ProbRead's efficiency in trainable parameters, memory and speed. Its trainable-parameter budget is comparable to low-rank adaptation and substantially smaller than full fine-tuning, while both adaptation methods require substantially more peak GPU memory and training time in the measured settings.

The performance comparisons reported in the first-round rebuttal (A7 and A8) already show that ProbRead is competitive with full fine-tuning and low-rank adaptation in the evaluated settings. On Caco-2, the corresponding performance and cost measurements show competitive prediction with substantially lower memory use and faster training, supporting a favorable accuracy-cost trade-off. The additional S-PLM EC measurements show that the resource savings also extend to a larger protein model.

These comparisons address complementary questions. The capacity-matched MHAP comparison directly evaluates the readout contribution with frozen representations and matched trainable capacity. Full fine-tuning and low-rank adaptation assess the broader accuracy-cost trade-off when updating the backbone. A higher task metric from either adaptation method must therefore be weighed against its additional resource cost when assessing the value of ProbRead's efficient frozen readout.

We understand the value of extending these complementary comparisons consistently across all task groups to provide a more complete assessment of both the readout contribution and the accuracy-cost trade-off. We cannot complete the full performance matrix for full fine-tuning and low-rank adaptation within the rebuttal period. We will add these additional experiments in the final version, providing a unified comparison of MHAP, low-rank adaptation and full fine-tuning across the task groups.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.