acceptodds
Withdrawn from ICLR 2027

Learning to Read Protein Language Models: Input-Conditioned Fusion of Layerwise Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Protein language models (PLMs) learn structure- and function-related representations through pretraining, and recent advances in structure-aware and multimodal modeling have further enriched their representational capabilities. However, existing research primarily focuses on improving representations through pretraining or encoder enhancement, while downstream applications typically rely on the final layer or fixed layer readout strategies, with limited systematic investigation into how to fully exploit existing multilayer information without modifying the pretrained encoder. Through layerwise evaluation, we find that the final layer is not the optimal source of representations for multiple downstream tasks, and that effective readout locations vary across models and tasks, indicating the need to distinguish pretrained models' representational capabilities from their downstream readout strategies. Based on this observation, we propose a protein-specific adaptive layer fusion method, which learns a mapping from multilayer protein representations to layer weights using supervision from the target task, producing corresponding fused representations for different inputs while keeping the pretrained encoder frozen. Our evaluation across multiple representative PLMs, three datasets and four tasks shows that reusing multilayer representations can improve over final-layer readout, and when applied to base PLMs, our method can outperform models improved through encoder enhancement. As our findings indicate, the downstream capabilities of PLMs depend not only on what representations are learned, but also on how these representations are read out for the target task, with representation learning and representation readout constituting two interrelated research dimensions.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Language Models, Protein Representation Learning, Input-Conditioned Layer Fusion, Representation Adaptation, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rKDbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to combine representations from different layers of pretrained PLMs. The authors propose DynamicGate, which learns protein-specific layer weights using downstream task supervision while keeping the pretrained encoder frozen. Experiments across multiple PLMs and four downstream tasks show improvements over fixed-layer and shared-weight fusion baselines, although the gains are not consistent across all settings.

Critical strengths
  • DynamicGate improves downstream performance across several PLMs and tasks without fine-tuning the pretrained encoders.
  • The experiments show that useful layer representations vary depending on the downstream task, suggesting that how pretrained representations are combined also matters.
Critical weaknesses
  • The advantage of protein-specific weighting over shared layer fusion is less clear on some tasks. While DynamicGate performs better on PISCES, Mean-Gate achieves comparable or slightly better results on several SCOP classification settings (App. D.1).
  • DynamicGate introduces additional parameters compared with simpler baselines. Although App. D.2 includes parameter-matched MLP controls, comparing training and validation performance would help clarify whether there are generalization gaps.
  • Some methodological details need clarification. Sec. 2.2 does not specify the architecture of the shared scoring network, and Top-4/8 Gate is introduced in Sec. 4.3 without a definition.
  • The related work could also cover studies on intermediate representations and layer fusion in vision and NLP, beyond PLMs.
Questions for authors
  • Could the authors include Mean-Gate as a baseline in the main result tables, particularly Tables 2 and 3, to clarify the benefits of protein-specific weighting over shared layer fusion?
  • Could the authors report training and validation performance for different fusion strategies to help assess potential generalization gaps?
Reviewer_MrcbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose that use input-dependent layer weights for each protein and fuses representations from multiple layers to support structural similarity and classification tasks. The central question is how to make full use of the information already available in pretrained models.

Critical strengths
  • The paper evaluates the encoder's representational capacity separately from the downstream readout strategy, addressing a practically relevant question. The method is simple and requires no updates to the pretrained encoder.
  • On PISCES, all seven main PLMs outperform the static baselines listed in the main text. For example, the Spearman correlation for ESM-2 650M increases from 0.6587 for the strongest static baseline to 0.7158.
  • The experiments with Mean-Gate trained from scratch, weight permutation, and transfer are informative. The authors also report settings in which dynamic fusion does not perform better.
Critical weaknesses
  • The methodological novelty is limited. Input-dependent layer fusion has precedents in language models, such as Attention Fusion (2022). The main changes in this paper concern the application domain, pooling granularity, and scoring function. A direct comparison with existing linear dynamic layer attention is missing, leaving it unclear how much additional capability the MLP provides.

  • The added value of dynamic weighting has not been sufficiently isolated. In Appendix Table D.2, the SCOP Fold accuracy for ESM-2 650M is "Mean-Gate 36.72±1.01%, Dynamic Gate 36.21±1.24%," which does not demonstrate an advantage for dynamic weights. Moreover, the MLP control is reported only on SCOP. PISCES still lacks a baseline combining "static fusion + supervised projection + cosine similarity," making it difficult to rule out gains from general nonlinear supervised adaptation.

  • The evidence for generalization and statistical reliability remains limited. The PISCES fusion test set contains only 460 pairs involving shared proteins; CATH and EC each contain only 165 test proteins and use coarse-grained labels. PISCES, CATH, and EC do not use splits that separate internal homology clusters, and paired confidence intervals comparing the dynamic approach with the strongest static approach are also missing. These limitations constrain the assessment of the stability of the gains and generalization to remote homologs.

Questions for authors
  • Under the same feature, hyperparameter-tuning, and model conditions, can linear dynamic attention or static fusion with supervised projection match Dynamic Gate?
  • Can the advantage over the strongest static baseline be maintained across repeated homology-separated splits and verified using confidence intervals that account for correlations induced by shared proteins?
  • How do the authors ensure that there is no data leakage between the pretraining data and the downstream task datasets?
Reviewer_SjRxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DynamicGate, which learns weights over layerwise representations using a scoring network. The authors observe that intermediate-layer representations often exhibit stronger correlations with protein structural similarity than final-layer representations. Experiments across multiple PLMs and downstream tasks demonstrate the benefits of multilayer representation fusion

Critical strengths
  1. The paper addresses a relevant problem in PLM representation utilization, which may also provide insights into the non-monotonic scaling behavior observed in downstream protein prediction tasks. Previous studies have reported that increasing model size or extending pretraining does not consistently translate into improved downstream performance. In this context, investigating how task-relevant information is distributed and utilized across PLM layers is particularly meaningful.

  2. The results suggest that representation readout strategies may influence the apparent scaling behavior of PLMs, offering a useful perspective. The experimental evaluation covers several pretrained models with different architectures and training strategies.

Critical weaknesses
  1. limited methodological novelty

The proposed DynamicGate is essentially an input-conditioned extension of existing layerwise weighted aggregation methods. Learnable layer fusion has been extensively explored in representation learning, including ELMo and ESMFold, while previous studies have already investigated the distribution of structural and functional information across PLM layers.

  1. the real benifit of protein-specific dynamic fusion

This is my primary concern. While the experiments consistently demonstrate the benefits of multilayer representation aggregation, the gains attributable to dynamic fusion are not consistent. In Table 2, DynamicGate outperforms Global Scalar Mix in only 7 of the 12 SCOP experiments, with several differences being relatively small. Moreover, appendix Table D.2 shows that Mean-Gate achieves comparable or better performance than DynamicGate in most SCOP settings. These results suggest that much of the improvement is from multilayer aggregation, while the contribution of dynamic fusion remains uncertain. Overall, the paper provides strong support for multilayer representation fusion in general than for the proposed DynamicGate mechanism specifically.

  1. incomplete comparison with other FT strategies and computational efficiency

The evaluation focuses primarily on different readout methods applied to frozen PLMs. Existing studies have also explored supervised representation learning, parameter-efficient fine-tuning, and partial/full fine-tuning for PLMs. The computational efficiency of the proposed method also deserves a more quantitative evaluation. For instance, consider ESM-2 650M, DynamicGate scoring introduces approximately 82K parameters (MLP 1280-64-1), and classification head adds over 1M parameters. In additional, DynamicGate must also retain or extract representations from all layers. What's more, the ESM-2 and S-PLM experiments employ different pooling procedures. A controlled experiment would strengthen the comparison of proposed method with others.

Questions for authors

All my concerns are listed in the Weaknesses section, including novelty argument, necessity of dynamic fusion compared with simple mean fusion, and controlled comparison with other settings/methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.