acceptodds
Under review as a conference paper at ICLR 2027

MCRA: Multi-Context Residue Adaptation for Protein Language Model Transfer

Abstract

Protein language models encode complementary information across layers and sequence contexts, but frozen-backbone transfer often relies on the final layer or a uniform adapter. We introduce MCRA (Multi-Context Residue Adaptation), a residue-wise multi-scale framework that learns task-specific layer fusion, routes residues among context experts, and gates the adapted features with the frozen final-layer representation. On 13 protein prediction benchmarks, we compare MCRA with eight frozen protein models equipped with independently trained six-layer Transformer adapters, using the same data splits and evaluation metrics. MCRA obtains the highest or tied-highest score on 11 tasks. Component comparisons examine layer fusion and context adaptation, while perturbation analyses describe task-dependent changes in sequence-averaged routing. Together, these results support coordinating task-level depth selection with residue-level context adaptation for frozen protein model transfer.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Language Models, Parameter-Efficient Transfer Learning, Multi-Scale Adaptation, Residue-Wise Routing, Layer Fusion, Protein Representation Learning, Protein Function Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GjtQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce a method)MCRA) to adapt protein language models for downstream tasks. MCRA learns a weight for each layer, and the weights are shared within a task and it routes each residue to 8 experts with different local convolution settings and picks the top four. It also create a gate combines the adapted features with the frozen backbone's final output. The authors test MCRA on 13 tasks and the results show it promising performance.

Critical strengths
  1. The problem is clear, and the design fits it. Layer fusion handles differences between tasks. Expert routing handles differences between residues. The gate controls how much of the pretrained representation is kept. The backbone stays frozen, so the method is simple and easy to apply.

  2. The evaluation is broad and the gains are clear. The paper covers 13 tasks, several backbones, and multiple adaptation methods. Contact prediction gains 0.0852 over the strongest baseline and 0.1338 over the same-backbone Transformer adapter. The gain also shows up across other backbones.

  3. The baselines are well chosen, and the paper is honest about its limits. The main baseline is a supervised Transformer adapter. The TF-Mix variant keeps the same expert pool and top-4 rule but removes input-dependent routing, so it directly tests whether routing matters. The paper also reports Head-only, T-Adapter, and Full results separately. It states clearly that layer weights, scale statistics, t-SNE, and gate weights are only descriptive evidence

Critical weaknesses
  • Statistical reliability is weak. The main 13-task results and ablations use one seed only. There are no repeated-run means, standard deviations, or confidence intervals.
  • The contribution of each module is not isolated. TF-Mix shows that input-dependent routing helps, but there is no protein-level routing control to test the residue-level gain. Layer fusion lacks a uniform-averaging control.
  • The efficiency claims lack execution detail. In a standard MoE implementation, all experts are computed before top-4 weighting, so top-4 does not cut expert compute in half.
  • Although the authors have conducted substantial and rigorous work, the conclusions lack novelty, as they have already been repeatedly demonstrated and validated in previous studies. The present work merely extends these well-established findings to the context of proteins.
Questions for authors
  • Do the gains hold across repeated runs with multiple seeds?
8 comments

Response to Q1:

We fully understand and agree with your concern about the statistical variance of single-seed results. Before submission, due to the large computational cost of hundreds of fine-tuning experiments, we were unable to complete repeated runs with multiple random seeds, and we acknowledged this limitation.

During the rebuttal period, we have now completed three-seed evaluations across all 13 tasks for MCRA, MCRA-Small, all 8 PLM baselines, and all ablation variants. We have uploaded the revised PDF with all new results and text updates highlighted in blue in Tables 1–2 and Tables 5–6, as well as in Tables G1–G3 of the General Comment above.


Response to W2:

Thank you very much for your careful consideration of the granularity of our ablation experiments. Based on the quantitative results in the main paper and appendices, we explain the independent contributions of residue-wise routing and learnable layer fusion compared with their coarse-grained alternatives.

1. Independent Evidence for the Benefits of Residue-Wise Routing over Protein-Level Routing

Although Tables 5–6 and Eq. 19 in Appendix B.11 directly examine input dependence through TF-Mix, two quantitative findings in our paper further separate the contribution of residue-level granularity from protein-level input-dependent routing.

Evidence A: Residue-Level Scale Differences Across 3D Structural Microenvironments Within a Single Protein

By mathematical definition, protein-level input-dependent routing can generate only one weight vector for a given protein sequence, forcing all residues in the protein to use the exact same mixture of convolutional kernel scales.

However, in our post-hoc structural alignment analysis of TEM-1 β\beta-lactamase in Section 4.2 and Appendix D, supported by Figs. 6 and 10, Table 19, and Eq. 30, we measured routing probability distributions across different structural regions within the same protein chain:

  • Secondary structures: As shown in the middle panel of Fig. 10, α\alpha-helix and β\beta-strand residues within the same protein assign average probabilities of 0.22 and 0.17 to the Medium scale, respectively, while random coil/loop regions assign only 0.07. Meanwhile, loop regions assign a probability of 0.85 to the large receptive field scale, Large.
  • Solvent accessibility: As shown in the right panel of Fig. 10, residues buried in the hydrophobic protein core retain a Medium weight of 0.23 and a Small weight of 0.08. As residues transition through intermediate regions to the exposed hydrophilic surface, the Large scale weight increases monotonically from 0.68 to 0.84 and 0.88, while the Medium weight decreases from 0.23 to 0.09 and 0.08.

Protein-level routing cannot provide different context processing for the core and surface, or for helices and loops, within the same protein through its routing weights. Such residue-level adaptation to local microenvironments is especially important for Contact prediction, which directly relies on residue-pair representations in Eq. 14. On this task, MCRA achieves a substantial three-seed improvement of 0.0911 over T-Adapter with the same backbone, raising Top-L/5L/5 precision from 0.7555 to 0.8466—and up to 0.8500 for MCRA-Small—in Tables 1 and 5.

Evidence B: Residue-Wise Routing Outperforms TF-Mix Despite Stable Global Scale Distributions

Appendix C.5 and Eq. 26 define protein-level scale distributions by averaging routing probabilities across residues. Table 18 and Fig. 2 show nearly constant global distributions for DeepSol, Human-PPI, Sor. Sig., Clo. CLF, and DL Bin with Scale TV ≤0.0052\leq 0.0052, where one dominant scale approaches a probability of 1.0.

Nevertheless, Full MCRA outperforms TF-Mix on 11 of the 13 benchmarks in Tables 5–6, including consistent gains on tasks with nearly invariant global scale distributions such as DeepSol (0.8340 vs. 0.8141), Clo. CLF (0.8413 vs. 0.8306), M. I. Bin. (0.7437 vs. 0.7252), DL Bin (0.9426 vs. 0.9407), and Sor. Sig. (0.8640 vs. 0.8630).

These results suggest that stable global scale distributions do not necessarily imply constant residue-level routing. Sequence averaging may conceal local routing variations and expert switching within the same scale group, as noted in Appendices C.5 and E.1. Residue-wise routing can capture position-specific features and cooperate with the residue-level gate in Eq. 13, although these comparisons do not isolate routing granularity from input adaptivity.

2. Learnable Layer Fusion: Mechanisms and Supporting Evidence

  • Task-Specific Layer Preferences: MCRA initializes the last 16 layer-fusion weights uniformly at 1/161/16, as specified in Table 10 and Eq. 33. However, Fig. 4 reveals distinct learned preferences: Contact emphasizes layer 18 at approximately 3×3\times the uniform weight, Fold Cls favors layers 22–25, and DeepSol and Human-PPI emphasize layers 26–32. These deviations indicate that different tasks favor different representation depths.
  • Performance Evidence: DeepSoluE exhibits relatively uniform layer weights and shows only a modest three-seed improvement over w/o Layers at 0.7647 vs. 0.7628. In contrast, tasks with stronger layer preferences achieve substantially larger gains in Tables 5–6: Fold Cls improves from 0.6644 to 0.8065, Anti. Res. from 0.6126 to 0.7050, M. I. Bin. from 0.6862 to 0.7437, and Sor. Sig. from 0.8425 to 0.8640.

These observations align with prior findings that different pretrained layers encode complementary biological information [1–3]. Learnable fusion enables task-specific selection of representation depths using only 16 scalar weights in Eqs. 3–4.


Response to W3:

Thank you for pointing out this important implementation detail. Your observation is completely correct: in an MoE implementation with sequence-context operators, Top-4 weighting does not halve the expert computation. In fact, we discussed this issue in detail in Appendix B.9 and Eq. 18 and reached exactly the same conclusion. We apologize that this discussion was previously confined to the appendix, and we have now explicitly incorporated this explanation into Section 4.4 (highlighted in blue in the revised PDF).

1. Why Top-4 Routing Does Not Halve Active Parameters or Expert Computation

  • As described in Section 3.3 and Eqs. 6–8, unlike standard MLP mixture-of-experts models [1, 2], which apply linear transformations independently to individual tokens, each MCRA expert includes local depthwise convolutions and full-sequence multi-head self-attention.
  • As explained in Appendix B.9, the standard MoE implementation evaluates all experts before Top-4 weighted fusion, while our sparsity analyzer provides a sequence-level token_union dispatch mode. To preserve the convolutional neighborhood and full-sequence self-attention context of each residue, individual residues cannot be separated from the sequence and processed independently. In token_union mode, the system takes the union of the Top-4 experts selected by all valid residues in each sequence, skipping an expert only when no residue in that sequence selects it.
  • According to the sequence-dispatch parameter counting rule in Eq. 18 of Appendix B.9, the active parameters per sample include all non-expert parameters plus expert parameters weighted by their sequence-level dispatch rates. Since the union of Top-4 selections across different residues in a single sequence usually activates more than four experts, Table 7 reports 718.6M active parameters per sample, accounting for 92.41% of the total 777.6M parameters rather than 50%. As stated in Appendix B.9, Top-4 routing alone therefore does not imply half the expert FLOPs or a particular latency reduction.

2. The Actual Source of MCRA's Measured Speedup over Capacity-Matched LoRA and DoRA in Table 7

  • As shown in Table 7 and Appendices B.8–B.9, under closely matched trainable parameter budgets of approximately 122M–125M, MCRA requires 1041.9 GFLOPs for forward computation—slightly less than 1057.9 GFLOPs for high-rank LoRA and 1113.4 GFLOPs for DoRA. Meanwhile, MCRA achieves a measured inference latency of 24.21±0.7224.21 \pm 0.72 ms per sample, which is 1.83× faster than LoRA at 44.24±0.8744.24 \pm 0.87 ms and 2.17× faster than DoRA at 52.46±1.2752.46 \pm 1.27 ms.
  • This measured latency advantage does not come from halving expert computation. Instead, it stems from the architectural execution differences illustrated in Fig. 1a and Table 13. Under the same parameter budget, LoRA and DoRA must perform low-rank updates or weight decomposition in the Query and Value linear modules across all 33 Transformer layers of ESM2-650M. In contrast, MCRA keeps all 33 layers of the ESM2 backbone fully frozen without inserting any internal operators, performing only a single weighted sum of the hidden states from the last 16 layers and executing adaptation in a parallel side branch.

Response to W4:

We fully respect your perspective. Indeed, cross-layer weighting, grouped mixture attention [1], and sparse mixture-of-experts routing [2, 3] are established basic operators in general computer vision and natural language processing. However, this work is not a simple transfer of these general components to the protein domain. Instead, we propose a joint adaptation architecture to address the coexistence of depth heterogeneity and sequence-context heterogeneity in downstream transfer from frozen protein language models. Through experiments and analyses, we reveal three domain-specific findings:

1. We demonstrate task-dependent preferences for representation depth and context scale in protein transfer, with ablation results supporting their complementary roles.

Previous studies on protein transfer either mainly investigated the attention mechanisms or representation properties of different pretrained layers [4, 5], applied uniform fine-tuning operators to fixed layers [6, 7, 8], or used multi-scale feature fusion with fixed depths [9].

By jointly learning task-level layer fusion in Eqs. 3–4 and residue-level scale routing in Eqs. 6–11, Figs. 2 and 4 show that different protein tasks have distinct preferences along two complementary dimensions: representation depth and local context scale.

As described in Section 4.2, even among structure-related tasks, Contact prefers the earliest selected layer—with layer 18 receiving the highest weight—combined with the Medium convolutional receptive field, while Fold Cls mainly uses early-to-intermediate layers 22–25 together with the XLarge receptive field.

Similarly, among whole-protein property tasks, DeepSol prefers deeper intermediate layers 27–32—peaking at layer 29—and the XLarge receptive field, while Human-PPI relies on intermediate layers 26–32—peaking at layer 26—and the Small receptive field.

The three-seed ablation experiments in Tables 5–6 further confirm that the full model, which retains both learnable layer fusion and residue-level adaptive routing, outperforms the final-layer-only variant w/o Layers on all 13 tasks and surpasses the task-shared routing variant TF-Mix on 11 of 13 tasks.

2. We observe a post-hoc association between residue-level routing and local 3D structural environments using sequence-only inputs.

As described in Appendix A.1 and Tables 8 and 10, MCRA uses only amino acid sequences during training and inference, without 3D structural coordinates, contact maps, DSSP secondary structure labels, or solvent accessibility as auxiliary inputs.

In the single-protein post-hoc structural alignment analysis of TEM-1 β\beta-lactamase in Fig. 6, Fig. 10, and Table 19 of Appendix D, residue-level routing shows scale adjustments associated with local structural microenvironments, despite an overall preference for the Large scale.

Ordered α\alpha-helix and β\beta-strand regions retain higher proportions of Medium scale routing than random coil/loop regions, with average probabilities of 0.22 and 0.17 compared with 0.07 for loop regions.

Meanwhile, solvent-buried residues assign a probability of 0.23 to the Medium scale. For intermediate and exposed surface residues, the Large scale probability increases from 0.68 to 0.84 and 0.88, while the Medium probability decreases to 0.09 and 0.08.

As carefully stated in Appendix D, this illustrative case suggests that residue-level routing can capture context-scale cues associated with local structural environments from sequence alone, even without structural supervision.

Reviewer_T3cYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MCRA, a framework for adapting frozen protein language models to downstream tasks. It learns task-specific weights to fuse hidden states across PLM layers, routes individual residues among context experts built with different convolutional kernels, and combines the adapted features with the backbone's final-layer representation through a gating mechanism. MCRA comes out best or tied-best on 11 of the 13 tasks, and it improves 17 of the 18 additional task-backbone combinations. The paper also compares against other adaptation methods, ablates the main components, and analyzes what the learned routing behavior looks like.

Critical strengths

1.Getting more transfer out of frozen PLMs is an important problem. Combining representations across layers with adaptive context processing strikes me as a sensible direction, and the framework accommodates several kinds of protein prediction tasks.

2.The evaluation spans 13 tasks plus six additional PLM backbones, with a particularly large gain on Contact prediction. Table 4 also shows consistent improvements over TF-Mix and the variant without layer fusion, which supports the value of input-dependent routing and learned layer fusion.

Critical weaknesses

1.The main comparisons do not really control for model capacity or training budget. MCRA has 125.3M trainable parameters; the parameter counts of the Transformer-adapter baselines are never reported, and the batch sizes differ as well. The LoRA/DoRA comparison only covers three tasks. Without capacity-matched baselines and multi-seed runs, it is hard to tell whether the gains come from MCRA's design rather than differences in capacity or training.

2.MCRA largely assembles existing techniques — layer fusion, multi-scale attention, expert routing — into one pipeline. The ablations do support layer fusion and input-dependent routing, but they do not fully establish the benefits of the specific design. In particular, it remains unclear whether experts with different kernel scales actually beat identical-kernel experts, or whether residue-wise routing has any advantage over routing at the protein level. As a result, the added value of the proposed combination is difficult to assess.

Questions for authors
  1. Could you report the trainable parameter counts and training compute of the Transformer-adapter baselines? Would MCRA still beat a simpler adapter under a comparable parameter or compute budget?
  2. Does using different kernel scales across experts actually help, compared with giving all experts the same kernel configuration? The fixed-scale ablation also changes model capacity, so this is hard to tell from the current results.
  3. The TF-Mix comparison supports input-dependent routing, but it does not isolate residue-wise routing itself. Do you have evidence that residue-wise routing beats protein-level input-dependent routing?
5 comments

Response to Q1:

We sincerely thank you for recognizing the clarity of our paper and appreciating our approach of combining cross-layer representation fusion with adaptive context routing to address the challenges of frozen PLM transfer. We respectfully respond to your key questions as follows:

  1. Complete Capacity-Matched Multi-Seed Results (~18.2M): The Transformer-Adapter baseline has 18.23M trainable parameters (specifically 18.2261M with a binary head). To ensure strict capacity matching, we designed MCRA-Small with 18.21M trainable parameters. We have now completed three-seed evaluations across all 13 tasks for MCRA, MCRA-Small, all PLM baselines, and all ablation variants. These complete results are reported in Table G1 of the General Comment, as well as in Tables 1–2 and Tables 5–6 of the revised PDF, with all updates highlighted in blue.
  2. Rigorous Capacity-Matched Comparisons Already Included in the Paper (~122M–125M): In addition, in Tables 4 and 7 of the main paper and Appendix B.8, we conducted controlled comparisons with mainstream parameter-efficient fine-tuning methods under closely matched trainable parameter budgets of ~122M–125M:
  • LoRA: 122.1M trainable parameters;
  • DoRA: 123.2M trainable parameters;
  • MCRA with 8 experts and Top-4 routing: 125.3M trainable parameters, as detailed in Table 7.

Under this strictly capacity-matched setting in Table 4, MCRA significantly outperforms high-budget LoRA and DoRA on all three tested tasks across distinct categories:

  • Human-PPI for interaction prediction: MCRA 0.8480 vs. LoRA 0.8210 and DoRA 0.8062;
  • Anti. Res. for functional classification: MCRA 0.7168 vs. LoRA 0.6313 and DoRA 0.6195;
  • Sor. Sig. for sorting signal prediction: MCRA 0.8830 vs. LoRA 0.8677 and DoRA 0.8724.

These results directly demonstrate that even with the same high-capacity budget of 122M–123M parameters, existing mainstream fine-tuning methods still perform clearly worse than MCRA. This indicates that the improvements do not come from simply increasing the parameter count. 3. In summary, controlled experiments consistently show that MCRA's performance gains come from cross-layer fusion and multi-scale residue routing mechanisms, both in the lightweight setting of approximately 18.2M parameters and in the high-capacity setting of approximately 125M parameters. All complete MCRA-Small results and multi-seed analyses are now included in the revised manuscript and highlighted in blue.


Response to Q2:

We thank the reviewer for pointing out model capacity as a confounding factor. To further examine whether the performance gains are driven only by parameter count, we added MCRA-Small, which has 18.21M trainable parameters—matching the 18.23M ESM2 T-Adapter baseline—and maintains comparable performance to the original MCRA. Under identical training and evaluation protocols across three random seeds, MCRA-Small uses substantially fewer parameters than the 67.6M fixed-scale operator (w/o Adapt.) while achieving superior performance across all 13 tasks, as shown in Tables 1–2, Tables 5–6, and the General Comment above (highlighted in blue in the revised PDF). This confirms that increasing model capacity alone cannot explain the observed gains.

Together with the task-level scale preferences in Fig. 2 and the residue-level routing analysis in Fig. 6, these results further support the value of the adaptive multi-scale design.



Response to Q3:

Thank you very much for this insightful question! Our structural microenvironment analysis of a single protein in Appendix D and cross-task routing variance analysis across Table 18, Fig. 2, and Tables 5–6 provide strong direct and indirect evidence that residue-wise routing has fundamental advantages over protein-level input-dependent routing.

1. Direct Evidence: Residue-Level Scale Adaptation to Different 3D Structural Microenvironments Within a Single Protein

By definition, protein-level input-dependent routing can only produce one routing weight vector p(x)p(x) for a given protein sequence xx, forcing all residues i∈{1,…,L}i \in \{1,\dots,L\} within that protein to share the same scale mixture weights.

However, in our post-hoc structural alignment analysis of TEM-1 β\beta-lactamase (UniProt P62593, PDB 1BTL), presented in Section 4.2 and Appendix D alongside Figs. 6 and 10, Table 19, and Eq. 30, we found that MCRA—using only the amino acid sequence as input—shows dynamic scale adaptation closely associated with local 3D structures at different residue positions within the same protein:

  • Differences between secondary structures within the same protein: As shown in Fig. 10, ordered α\alpha-helix and β\beta-strand residues in TEM-1 assign average routing probabilities of 0.22 and 0.17 to the medium receptive field Medium (3,7,15), respectively. These values are 3.1 and 2.4 times those of random coil/loop regions, whereas flexible loop regions increase the weight of Large (5,9,17) to 0.85.
  • Gradual changes in solvent accessibility within the same protein: As detailed in Fig. 10 and Appendix D, when residues transition from the buried hydrophobic core (RSA≤0.20\text{RSA}\leq 0.20) through intermediate regions to the exposed hydrophilic surface (RSA>0.50\text{RSA}>0.50), the residue-wise router shows a clear monotonic shift in scale allocation. Buried core residues retain a combined 0.31 routing weight for medium and small local receptive fields (Medium at 0.23 plus Small at 0.08). In contrast, exposed surface residues show a monotonic increase in Large scale probability from 0.68 to 0.88, while Medium decreases from 0.23 to 0.08.
  • Conclusion: Because protein-level routing assigns the same weights to the entire sequence, it cannot distinguish helices from loops or buried cores from exposed surfaces within the same protein through its routing weights. Residue-level granularity is essential for tasks that strongly depend on local structural differences, such as Contact prediction, which directly uses residue-pair representations (Fi,Fj)(F_i,F_j) to predict a 2D contact map in Eq. 14. On this task, MCRA achieves a substantial improvement over the same-backbone T-Adapter, raising three-seed Top-L/5L/5 precision from 0.7555 to 0.8466 (+12.06%, and 0.8500 for MCRA-Small) in Tables 1 and 5.

2. Indirect Evidence: Residue-Wise Full Still Significantly Outperforms Task-Shared TF-Mix Even When Sequence-Level Average Routing Distributions Are Nearly Constant

Combining Eq. 26 in Appendix C.5,

pg(x)=1Lx∑i=1Lx∑e∈Egri,e(x)p_g(x)=\frac{1}{L_x}\sum_{i=1}^{L_x}\sum_{e\in\mathcal{E}_g}r_{i,e}(x)

with Table 18 and Fig. 2 reveals an important observation:

  • On tasks such as DeepSol, Human-PPI, Sor. Sig., M. I. Bin., DL Bin, and Clo. CLF, when we average residue-level routing probabilities across the entire protein sequence to obtain the protein-level scale distribution, the dominant scale remains almost constant across samples and perturbations. The global sequence-averaged shift Scale TV is ≤0.0052\leq 0.0052 for these tasks in Table 18, and the proportion of a single dominant scale is close to 1.0 in Fig. 2.
  • This means that on these tasks, a protein-level router that determines scale allocation only from sequence-averaged features would learn a sequence-level scale distribution highly similar to the static task-level mixture TF-Mix in Eq. 19, effectively converging to the same globally dominant scale.
  • However, the three-seed ablations in Tables 5–6 show that residue-wise routing (Full) still outperforms TF-Mix on 11 of the 13 tasks overall, including consistent gains on tasks where the protein-level average scale distribution barely changes—such as DeepSol (0.8340 vs. 0.8141), M. I. Bin. (0.7437 vs. 0.7252), Clo. CLF (0.8413 vs. 0.8306), and DL Bin (0.9426 vs. 0.9407).
  • Why can residue-wise routing still deliver clear gains when the average scale distribution across the entire protein barely changes? As explained in Appendix C.5 and Appendix E.1, there are two reasons:
  1. Sequence averaging in Eq. 26 can hide local scale changes at a small number of important functional sites within the sequence, such as short sorting signal peptides in Sor. Sig. (discussed in Appendix B.10) or local Small/Medium regions in TEM-1 (Fig. 10).
  2. Each scale group Eg\mathcal{E}_g contains two independently parameterized experts. Residue-wise routing can assign fine-grained weights to these two experts based on different amino acid microenvironments, while working alongside the residue-level gate βi=σ(Wf[Ei;M~i])\beta_i=\sigma(W_f[E_i;\tilde{M}_i]) in Eq. 13 to fuse features at each residue position.

This comparison provides strong evidence that the performance gains come from local dynamic allocation at the residue level, rather than coarse-grained weighting at the protein level.


Reviewer_KEVsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Summary

The paper proposes MCRA for adapting frozen protein language models to downstream tasks. It combines features from different backbone layers, uses residue-level routing to select local sequence contexts, and merges the adapted features with the original representations.

Contributions

The main contribution is an architecture combining layer selection, adaptive context routing, and gated feature fusion. The paper also studies how layer and context preferences vary across tasks and residues.

Validation

The method is evaluated across diverse protein prediction tasks and multiple pretrained backbones. Experiments include comparisons with existing adapters and fine-tuning methods, component ablations, and analyses of learned routing behavior.

Critical strengths
  1. Broad evaluation with improvements across backbones.

The evaluation covers 13 tasks and 8 frozen PLMs, with improvements in 17 of 18 comparisons across six additional backbones. This provides useful evidence that the improvements extend beyond a single model.

  1. Transparent limitations.

The appendix acknowledges relevant limitations, including data splits and routing behavior. This transparency is appreciated and makes the results easier to assess.

Critical weaknesses
  1. The main comparison lacks capacity-matched controls.

MCRA trains 125.3M parameters compared with approximately 17M for the baselines, without a matched adapter baseline or multiple seeds. This makes it difficult to attribute the improvements to the proposed mechanism rather than additional capacity. Since the individual components are largely established techniques in the field, stronger controlled comparisons would help clarify the contribution.

  1. The reported routing behavior raises questions about adaptivity.

The method emphasizes per-residue adaptive routing, but Table 16 reports routing shifts of ~0 on four tasks, with figure values near numerical precision. This raises questions about whether the router meaningfully adapts, potentially due to the entropy regularization (\lambda=1). Further analysis would help establish whether adaptive routing contributes to the improvements.

Questions for authors
  1. How many trainable parameters does the six-layer T-Adapter have? How does MCRA compare with a 125M-parameter adapter?
  2. Table 16 shows almost no routing changes on four tasks. Is the router effectively constant, and does lowering \lambda_ent help?
  3. Can the authors report results across 3–5 seeds, given the small differences in several ablations?
  4. Are the Contact and PPI prediction heads the same for MCRA and the baselines?
  5. Do the Fold Cls and Contact improvements hold under standard superfamily-disjoint splits?
7 comments

Response to Q1:

We sincerely thank the reviewer for highlighting the importance of controlling model capacity and reporting multi-seed results. We acknowledge that the main table in the original manuscript directly compared MCRA with a larger side-branch dimension against the standard 6-layer T-Adapter. This comparison did not fully separate the contributions of parameter size and architectural mechanisms. To address this issue, we developed a lightweight version, MCRA-Small, with a parameter count strictly matched to the baseline, and completed three-seed evaluations across all 13 tasks for MCRA, MCRA-Small, all PLM baselines, and all ablation variants. All new results and analyses have been incorporated into the revised PDF (highlighted in blue).

  1. Complete Capacity-Matched Multi-Seed Results (~18.2M): We present the complete capacity-matched comparison results in the General Comment above (Table G1) as well as in Tables 1–2 (page 6) and Tables 5–6 (page 9) of the revised PDF. Specifically, T-Adapter has 18.23M trainable parameters (18.2261M with a binary head), and MCRA-Small matches this budget at 18.21M trainable parameters. Across all 13 tasks and three random seeds, MCRA-Small consistently outperforms the capacity-matched ESM2 T-Adapter on every task.
  2. Rigorous Capacity-Matched Comparisons Already Included in the Paper (~122M–125M): In addition, in Tables 4 and 7 of the main paper and Appendix B.8, we conducted controlled comparisons with mainstream parameter-efficient fine-tuning (PEFT) methods under closely matched trainable parameter budgets (~122M–125M):
  • LoRA: 122.1M trainable parameters;
  • DoRA: 123.2M trainable parameters;
  • MCRA (8 experts, Top-4 routing): 125.3M trainable parameters (Table 7, page 10).

Under this strictly capacity-matched setting (Table 4, page 7), MCRA significantly outperforms high-budget LoRA and DoRA on all three tested tasks of different types:

TaskCategoryLoRADoRAMCRA
Human-PPIInteraction0.82100.80620.8480
Anti. Res.Functional Classification0.63130.61950.7168
Sor. Sig.Sorting Signal0.86770.87240.8830

These results directly show that even with the same high-capacity budget of 122M–123M parameters, existing mainstream fine-tuning methods still perform clearly worse than MCRA. This indicates that the improvements do not come from simply increasing the parameter count. Instead, they come from MCRA's explicit layer fusion and adaptive multi-scale routing mechanisms. 3. In summary, controlled experiments consistently show that MCRA's performance gains come from cross-layer fusion and multi-scale residue routing mechanisms, both in the lightweight setting of approximately 18.2M parameters and in the high-capacity setting of approximately 125M parameters. The complete MCRA-Small results and multi-seed analyses are now included in the revised manuscript (highlighted in blue).


Response to Q2:

We thank the reviewer for this insightful comment. The values in Table 18 (page 38) appear to be close to the numerical precision limit (e.g., 10−10∼10−2210^{-10}\sim 10^{-22}). However, the main reason is the mathematical definition of this metric: it measures a global average over the entire sequence, rather than the absence of residue-level responses.

1. Mathematical Meaning of Table 18: Global Averaging Cancels Out Local Dynamics

  • As defined in Appendix C.5 (Eqs. 26–27, page 37), Scale TV in Table 18 is the total variation distance between the global scale distributions of two sequences:

Dscale(x,x~)=12∑g∣pg(x)−pg(x~)∣D_{\text{scale}}(x,\tilde{x})=\frac{1}{2}\sum_g\vert{}p_g(x)-p_g(\tilde{x})\vert{}

where

pg(x)=1L∑i=1L∑e∈Egri,e(x)p_g(x)=\frac{1}{L}\sum_{i=1}^{L}\sum_{e\in\mathcal{E}_g}r_{i,e}(x)

represents the global scale distribution averaged over all valid residues in the entire sequence.

  • When we perturb a local window of 15–31 residues in a protein containing hundreds of residues, the local changes are averaged over the entire long sequence. As a result, the global total variation can naturally be diluted to extremely small values.

2. Why These Four Tasks (DeepSol, Human-PPI, DL Bin, Mat. Pro.)?

  • All four tasks are whole-protein sequence-level tasks.
  • As shown in Figure 2 (page 7 of the main paper), these tasks show a strong global scale inductive bias. Therefore, the model develops a form of macroscopic scale invariance (Macro-Invariance) across the entire sequence. However, this does not mean the model does not respond to perturbations. In contrast, the fusion gates of these tasks show strong responses in Table 18. For example, the β\beta shift reaches 0.1779 for DL Bin and 0.2252 for Clo. CLF.

3. The TF-Mix Ablation in Tables 5–6 Rules Out the Hypothesis That the Router Collapses to a Constant

  • If the router collapsed to a static assignment, MCRA Full should perform similarly to TF-Mix, which uses fixed task-level mixture weights.
  • As detailed in Appendix B.11, page 24, TF-Mix shares the same 8 experts, Top-4 sparsity, layer fusion, and gating mechanisms as MCRA Full, differing only in removing residue-level adaptive routing.
  • The three-seed results in Tables 5–6, page 9, show that MCRA Full outperforms TF-Mix on 11 of 13 benchmarks, with gains up to +4.17 pp. Full MCRA also wins on three of four tasks with near-zero Scale TV in Table 18. The exception, Human-PPI, has higher variance due to its 195-sample test set, although Full MCRA won on seed 2024 by 0.8480 vs. 0.8105:
TaskTF-Mix (3-seed mean)MCRA Full (3-seed mean)Difference (pp)
DeepSol0.8141±0.03510.8141 \pm 0.03510.8340±0.01300.8340 \pm 0.0130+1.99
DL Bin0.9407±0.00220.9407 \pm 0.00220.9426±0.00100.9426 \pm 0.0010+0.19
Mat. Pro.0.8248±0.00930.8248 \pm 0.00930.8277±0.00270.8277 \pm 0.0027+0.29
  • If the router lacked adaptivity, it would not be possible to achieve consistent improvements across 11 of 13 tasks. These results clearly confirm that residue-level dynamic routing cannot be replaced by static task-shared routing.

4. However, the Router Shows Strong Dynamic Adaptivity on Structural and Multi-Class Functional Tasks

For example, on Contact (Table 18, Fig. 9) and Fold Cls (Table 18, Fig. 12), perturbations cause substantial scale redistribution. Their Scale TV values reach 0.1413 and 0.1336, respectively, while their dominant-scale change rates (Dom. change) reach 38.34% and 10.43%.

In addition, the structural case study of TEM-1 β\beta-lactamase (Fig. 6, Fig. 10, Eq. 30) shows that routing probabilities vary significantly across residues within the same protein sequence, depending on secondary structure (α\alpha-helix: Medium = 0.22 vs. random coil Loop = 0.07) and solvent accessibility (buried residues: Medium = 0.23 vs. exposed residues: Medium decreases to 0.08, while Large increases from 0.68 to 0.88). These results demonstrate significant residue-level dynamic adaptation.

The entropy regularization coefficient λent=1.0\lambda_{\mathrm{ent}} = 1.0 was chosen to encourage sufficiently sharp selection probabilities across the Top-4 experts for each residue, preventing the expert weights from degenerating into a nearly uniform distribution. Meanwhile, we introduced a load-balancing coefficient λbal\lambda_{\mathrm{bal}} to strongly discourage all residues from collapsing onto a single expert (Appendix B.4, Eqs. 16–17, page 19).


Response to Q3:

We thank the reviewer for this comment. The original submission used one seed (2024) due to computational costs. We have now completed three-seed evaluations across all 13 tasks for MCRA, MCRA-Small, 8 PLM baselines, and all component ablations. The updated results in Tables 1–2 and 5–6, highlighted in blue, and General Comment Tables G1–G3 confirm low variance and consistent improvements over T-Adapter, Head only, w/o Layers, w/o Adapt., and TF-Mix.


Response to Q4:

Yes, they are exactly the same.

1. Contact

As described in Appendix B.2 and B.7 (pages 18, 20), all baselines and MCRA use exactly the same pairwise prediction head. Section 3.4 (page 5) of the main paper defines residue-pair prediction after Eq. (14) as:

y^ij=hpair(Fi,Fj).\hat{y}_{ij}=h_{\mathrm{pair}}(F_i,F_j).

Appendix B.2 states that Contact uses a pairwise prediction head and a masked pairwise binary objective. Appendix B.7 states that the T-Adapter baseline consists of a frozen PLM, a six-layer Transformer Adapter, and a task output module.

Based on the actual implementation, both methods take 480-dimensional residue representations as input. The prediction head can be written as:

uij=[Fi+Fj;,max⁡(Fi,Fj)]∈R960,u_{ij}=[F_i+F_j;\\,\max(F_i,F_j)]\in\mathbb{R}^{960},

ℓij=W2,ReLU(W1uij+b1)+b2,y^ij=σ(ℓij).\ell_{ij}=W_2\\,\mathrm{ReLU}(W_1u_{ij}+b_1)+b_2,\qquad \hat{y}_{ij}=\sigma(\ell_{ij}).

Here, the sum and maximum are computed element-wise, and the semicolon denotes concatenation. The MLP dimensions are 960→128→1960\rightarrow128\rightarrow1.

2. Human-PPI

Both the baseline and MCRA use exactly the same paired-sequence task interface. After sequence pooling, the representations are passed to the same classification head (Appendix B.2, page 18).

For protein pairs, Eq. (14) in the main paper can be written as:

zAB=∑tmtFtAB∑tmt,z_{AB}=\frac{\sum_t m_t F_t^{AB}}{\sum_t m_t},

Here, FABF^{AB} represents the concatenated and adapted representation of the protein pair, and mtm_t is the pooling mask. Training uses binary cross-entropy loss.


Response to Q5:

Thank you for the reviewer's reminder. As described in Appendix A.2 and Tables 8–9 (pages 16–17), to ensure consistency across all compared methods, we used the predefined dataset files provided by the public benchmarks [1,2]. We strictly applied a protein sequence similarity threshold of <30%<30\% across the training, validation, and test sets (approximately 8:1:1), without additionally constructing or validating superfamily-disjoint data splits.

MCRA uses multi-scale depthwise convolutions to capture low-level local physicochemical motifs (Fig. 1c, page 3). On Fold Cls and Contact, it also naturally assigns higher weights to earlier ESM2 layers that capture basic topological information (layers 18–25 in Fig. 4, page 8). These mechanisms suggest good potential for generalization to remote structural homologs.

We have clarified this evaluation scope in Appendix A.2 (highlighted in blue in the revised PDF) and noted strict superfamily-disjoint evaluation as an important direction for future work.

Thank you again for your valuable comments and careful review!

References

[1] Tim J. P. Hubbard, Bart Ailey, Steven E. Brenner, Alexey G. Murzin, and Cyrus Chothia. SCOP: a structural classification of proteins database. Nucleic Acids Research, 27(1):254–256, 1999.

[2] Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences (PNAS), 117(3):1496–1503, 2020.


Comments on the paper

2 comments

We sincerely thank all reviewers for their constructive feedback and valuable suggestions! We are very encouraged to see that the reviewers unanimously recognized our extensive experimental validation across 13 protein benchmark tasks and 8 frozen PLM backbones, as well as our candid discussion of method limitations in the appendix. In this general response, we centrally present the newly added controlled experiment tables (Tables G1–G3) and summarize the core points of clarification for reference in our subsequent point-by-point responses.

1. Model Capacity Matching Control: Newly Added Iso-Capacity MCRA-Small Multi-Seed Experiments

Regarding the concern about model capacity mismatch in the main table (125.3M vs. the baseline T-Adapter), we conducted verification from two dimensions: low-capacity alignment and high-capacity alignment:

  1. Low-capacity alignment around ~18.2M: In Appendix Table 10, the exact number of trainable parameters for the 6-layer T-Adapter baseline is 18.2261M. While keeping the original MCRA layer fusion, 8 multi-scale convolutional experts, Top-4 residue routing, and gating mechanism completely unchanged, we only reduced the side feature dimension side_gma_d_model in Equation 5 from 1920 to 672 to construct the lightweight version, MCRA-Small. Its trainable parameter count is 18.2067M, differing from T-Adapter by only 0.11%.
  2. High-capacity alignment around ~122M–125M: Tables 3, 4, and 5 in the main text already include three strictly iso-capacity high-budget baselines: LoRA (122.1M), DoRA (123.2M), and TF-Mix (123.3M), which retains the exact same 8-expert pool and layer fusion while removing dynamic routing. MCRA significantly outperforms them across all comparisons.

We have completed independent multi-seed training of MCRA-Small across all 13 tasks, with the results shown in Table G1:

Table G1: Comparison between iso-capacity MCRA-Small (18.21M) and the original MCRA (125.3M) across 13 tasks

TaskMetricMCRA-Small (18.21M)MCRA (125.3M)
DeepSolAUROC0.8444 ± 0.00170.8340 ± 0.0130
Fold ClsAcc.0.8045 ± 0.00250.8065 ± 0.0057
ContactTop-L/50.8500 ± 0.00080.8466 ± 0.0044
Human-PPIAUROC0.9108 ± 0.02320.8571 ± 0.0589
Anti. Res.Acc.0.6794 ± 0.02210.7050 ± 0.0156
Sor. Sig.F1-max0.8682 ± 0.00740.8640 ± 0.0172
Thermo.AUROC0.9701 ± 0.00010.9703 ± 0.0003
M. I. Bin.Acc.0.7433 ± 0.01670.7437 ± 0.0024
DL BinAUROC0.9411 ± 0.00260.9426 ± 0.0010
DL2 MultiF1-max0.7868 ± 0.00320.7876 ± 0.0050
Clo. CLFAUROC0.8427 ± 0.00230.8413 ± 0.0031
DeepSoluEAUROC0.7664 ± 0.01040.7647 ± 0.0012
Mat. Pro.Acc.0.8312 ± 0.00230.8277 ± 0.0027

2. Multi-Seed Statistical Reliability Verification

We sincerely accept the reviewers' criticism regarding the statistical rigor of using a single seed (seed 2024). Prior to submission, constrained by the massive training workload across 13 tasks and 8 backbone models, we were unable to complete multiple repeated runs in advance. Immediately after the rebuttal phase opened, we launched three-seed runs for the main table baselines and ablation variants. The three-seed results completed so far are summarized in Table G2 and Table G3.

Table G2: Completed three-seed results of main table baseline models

Task (Metric)VenusPLMESM-CProtGPT2PGLMDPLMProtSTESM3MCRA-Small (18.21M)MCRA (125.3M)
DeepSol (AUROC)0.6782 ± 0.13660.7458 ± 0.16140.7083 ± 0.13000.6858 ± 0.13140.6492 ± 0.15520.6528 ± 0.14470.6727 ± 0.11110.8444 ± 0.00170.8340 ± 0.0130
Fold Cls (Acc.)0.7209 ± 0.02920.6751 ± 0.04820.6286 ± 0.01620.7381 ± 0.02820.7748 ± 0.01700.5826 ± 0.12400.6325 ± 0.08030.8045 ± 0.00250.8065 ± 0.0057
Contact (Top-L/5)0.6731 ± 0.07240.7944 ± 0.06420.0441 ± 0.02360.7237 ± 0.07720.7903 ± 0.06360.4592 ± 0.05370.8574 ± 0.07900.8500 ± 0.00080.8466 ± 0.0044
Human-PPI (AUROC)0.8114 ± 0.01990.7630 ± 0.13860.7237 ± 0.13030.8411 ± 0.03310.8089 ± 0.04220.6325 ± 0.15320.6894 ± 0.05310.9108 ± 0.02320.8571 ± 0.0589
Anti. Res. (Acc.)0.6608 ± 0.01350.6736 ± 0.00170.6598 ± 0.02510.6785 ± 0.01280.6490 ± 0.03320.5624 ± 0.07540.6136 ± 0.02570.6794 ± 0.02210.7050 ± 0.0156
Sor. Sig. (F1-max)0.8371 ± 0.02420.8681 ± 0.01280.8150 ± 0.03300.8495 ± 0.03320.8599 ± 0.02180.8614 ± 0.01030.8225 ± 0.02420.8682 ± 0.00740.8640 ± 0.0172
Thermo. (AUROC)0.9013 ± 0.01700.9511 ± 0.00140.8831 ± 0.03680.9401 ± 0.00200.9483 ± 0.00770.9096 ± 0.03040.9044 ± 0.02600.9701 ± 0.00010.9703 ± 0.0003
M. I. Bin. (Acc.)0.7294 ± 0.02790.6941 ± 0.04030.6992 ± 0.02420.7484 ± 0.01210.7345 ± 0.03220.6054 ± 0.08080.7043 ± 0.04180.7433 ± 0.01670.7437 ± 0.0024
DL Bin (AUROC)0.9206 ± 0.00220.9094 ± 0.00400.9051 ± 0.00340.9176 ± 0.00440.9189 ± 0.01250.9299 ± 0.00930.9048 ± 0.00480.9411 ± 0.00260.9426 ± 0.0010
DL2 Multi (F1-max)0.7664 ± 0.02510.7824 ± 0.02470.7423 ± 0.03370.7713 ± 0.02040.7781 ± 0.01820.7744 ± 0.02210.7023 ± 0.03790.7868 ± 0.00320.7876 ± 0.0050
Clo. CLF (AUROC)0.7946 ± 0.03210.8026 ± 0.01730.7829 ± 0.00520.8159 ± 0.01840.7907 ± 0.02030.7111 ± 0.08400.7545 ± 0.01920.8427 ± 0.00230.8413 ± 0.0031
DeepSoluE (AUROC)0.6195 ± 0.14230.6319 ± 0.12840.6958 ± 0.01540.7138 ± 0.03920.7315 ± 0.00960.6114 ± 0.10260.6837 ± 0.02400.7664 ± 0.01040.7647 ± 0.0012
Mat. Pro. (Acc.)0.8063 ± 0.02020.8049 ± 0.01260.7761 ± 0.00750.7852 ± 0.03580.8095 ± 0.00700.7054 ± 0.02400.7997 ± 0.02160.8312 ± 0.00230.8277 ± 0.0027

Table G3: Completed three-seed results (Mean ± Std) for the ablation study

VariantDeepSolFold ClsContactHuman-PPIAnti. Res.Sor. Sig.Thermo.M. I. Bin.DL BinDL2 MultiClo. CLFDeepSoluEMat. Pro.
Full0.8340 ± 0.01300.8065 ± 0.00570.8466 ± 0.00440.8571 ± 0.05890.7050 ± 0.01560.8640 ± 0.01720.9703 ± 0.00030.7437 ± 0.00240.9426 ± 0.00100.7876 ± 0.00500.8413 ± 0.00310.7647 ± 0.00120.8277 ± 0.0027
T-Adapter0.7764 ± 0.09020.7387 ± 0.03200.7555 ± 0.03330.7817 ± 0.06560.6136 ± 0.04640.8548 ± 0.01550.9464 ± 0.01000.7391 ± 0.02420.9094 ± 0.00480.7802 ± 0.01600.7870 ± 0.03420.7133 ± 0.07310.8109 ± 0.0027
TF-Mix0.8141 ± 0.03510.7648 ± 0.02900.8236 ± 0.01810.8898 ± 0.06870.6873 ± 0.00590.8630 ± 0.01170.9694 ± 0.00070.7252 ± 0.01750.9407 ± 0.00220.7880 ± 0.01650.8306 ± 0.00480.7613 ± 0.00560.8248 ± 0.0093
w/o Adapt.0.8325 ± 0.02400.7611 ± 0.03950.8076 ± 0.00380.8030 ± 0.06200.6745 ± 0.01390.8509 ± 0.00770.9694 ± 0.00080.7349 ± 0.01060.9408 ± 0.00180.7732 ± 0.00810.8305 ± 0.00870.7610 ± 0.00560.8233 ± 0.0058
w/o Layers0.7980 ± 0.03590.6644 ± 0.10840.7980 ± 0.03660.8209 ± 0.00900.6126 ± 0.02370.8425 ± 0.00670.9684 ± 0.00080.6862 ± 0.01530.9341 ± 0.00260.7747 ± 0.00500.8242 ± 0.01280.7628 ± 0.00430.8173 ± 0.0037
Head only0.7370 ± 0.02120.2519 ± 0.24190.5156 ± 0.12980.7319 ± 0.05420.5221 ± 0.04130.5837 ± 0.10600.9515 ± 0.00760.4415 ± 0.09430.8377 ± 0.06850.6315 ± 0.07380.7413 ± 0.03910.7259 ± 0.02490.7585 ± 0.0220

Part of the model weights have already been anonymously uploaded to Hugging Face.Link:https://huggingface.co/mcra-anon-iclr27/mcra-weights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.