- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduce a method)MCRA) to adapt protein language models for downstream tasks. MCRA learns a weight for each layer, and the weights are shared within a task and it routes each residue to 8 experts with different local convolution settings and picks the top four. It also create a gate combines the adapted features with the frozen backbone's final output. The authors test MCRA on 13 tasks and the results show it promising performance.
Critical strengths
-
The problem is clear, and the design fits it. Layer fusion handles differences between tasks. Expert routing handles differences between residues. The gate controls how much of the pretrained representation is kept. The backbone stays frozen, so the method is simple and easy to apply.
-
The evaluation is broad and the gains are clear. The paper covers 13 tasks, several backbones, and multiple adaptation methods. Contact prediction gains 0.0852 over the strongest baseline and 0.1338 over the same-backbone Transformer adapter. The gain also shows up across other backbones.
-
The baselines are well chosen, and the paper is honest about its limits. The main baseline is a supervised Transformer adapter. The TF-Mix variant keeps the same expert pool and top-4 rule but removes input-dependent routing, so it directly tests whether routing matters. The paper also reports Head-only, T-Adapter, and Full results separately. It states clearly that layer weights, scale statistics, t-SNE, and gate weights are only descriptive evidence
Critical weaknesses
- Statistical reliability is weak. The main 13-task results and ablations use one seed only. There are no repeated-run means, standard deviations, or confidence intervals.
- The contribution of each module is not isolated. TF-Mix shows that input-dependent routing helps, but there is no protein-level routing control to test the residue-level gain. Layer fusion lacks a uniform-averaging control.
- The efficiency claims lack execution detail. In a standard MoE implementation, all experts are computed before top-4 weighting, so top-4 does not cut expert compute in half.
- Although the authors have conducted substantial and rigorous work, the conclusions lack novelty, as they have already been repeatedly demonstrated and validated in previous studies. The present work merely extends these well-established findings to the context of proteins.
Questions for authors
- Do the gains hold across repeated runs with multiple seeds?
8 comments
Evidence B: Residue-Wise Routing Outperforms TF-Mix Despite Stable Global Scale Distributions
Appendix C.5 and Eq. 26 define protein-level scale distributions by averaging routing probabilities across residues. Table 18 and Fig. 2 show nearly constant global distributions for DeepSol, Human-PPI, Sor. Sig., Clo. CLF, and DL Bin with Scale TV , where one dominant scale approaches a probability of 1.0.
Nevertheless, Full MCRA outperforms TF-Mix on 11 of the 13 benchmarks in Tables 5–6, including consistent gains on tasks with nearly invariant global scale distributions such as DeepSol (0.8340 vs. 0.8141), Clo. CLF (0.8413 vs. 0.8306), M. I. Bin. (0.7437 vs. 0.7252), DL Bin (0.9426 vs. 0.9407), and Sor. Sig. (0.8640 vs. 0.8630).
These results suggest that stable global scale distributions do not necessarily imply constant residue-level routing. Sequence averaging may conceal local routing variations and expert switching within the same scale group, as noted in Appendices C.5 and E.1. Residue-wise routing can capture position-specific features and cooperate with the residue-level gate in Eq. 13, although these comparisons do not isolate routing granularity from input adaptivity.
2. Learnable Layer Fusion: Mechanisms and Supporting Evidence
- Task-Specific Layer Preferences: MCRA initializes the last 16 layer-fusion weights uniformly at , as specified in Table 10 and Eq. 33. However, Fig. 4 reveals distinct learned preferences: Contact emphasizes layer 18 at approximately the uniform weight, Fold Cls favors layers 22–25, and DeepSol and Human-PPI emphasize layers 26–32. These deviations indicate that different tasks favor different representation depths.
- Performance Evidence: DeepSoluE exhibits relatively uniform layer weights and shows only a modest three-seed improvement over
w/o Layersat 0.7647 vs. 0.7628. In contrast, tasks with stronger layer preferences achieve substantially larger gains in Tables 5–6: Fold Cls improves from 0.6644 to 0.8065, Anti. Res. from 0.6126 to 0.7050, M. I. Bin. from 0.6862 to 0.7437, and Sor. Sig. from 0.8425 to 0.8640.
These observations align with prior findings that different pretrained layers encode complementary biological information [1–3]. Learnable fusion enables task-specific selection of representation depths using only 16 scalar weights in Eqs. 3–4.
Response to W3:
Thank you for pointing out this important implementation detail. Your observation is completely correct: in an MoE implementation with sequence-context operators, Top-4 weighting does not halve the expert computation. In fact, we discussed this issue in detail in Appendix B.9 and Eq. 18 and reached exactly the same conclusion. We apologize that this discussion was previously confined to the appendix, and we have now explicitly incorporated this explanation into Section 4.4 (highlighted in blue in the revised PDF).
1. Why Top-4 Routing Does Not Halve Active Parameters or Expert Computation
- As described in Section 3.3 and Eqs. 6–8, unlike standard MLP mixture-of-experts models [1, 2], which apply linear transformations independently to individual tokens, each MCRA expert includes local depthwise convolutions and full-sequence multi-head self-attention.
- As explained in Appendix B.9, the standard MoE implementation evaluates all experts before Top-4 weighted fusion, while our sparsity analyzer provides a sequence-level
token_uniondispatch mode. To preserve the convolutional neighborhood and full-sequence self-attention context of each residue, individual residues cannot be separated from the sequence and processed independently. Intoken_unionmode, the system takes the union of the Top-4 experts selected by all valid residues in each sequence, skipping an expert only when no residue in that sequence selects it. - According to the sequence-dispatch parameter counting rule in Eq. 18 of Appendix B.9, the active parameters per sample include all non-expert parameters plus expert parameters weighted by their sequence-level dispatch rates. Since the union of Top-4 selections across different residues in a single sequence usually activates more than four experts, Table 7 reports 718.6M active parameters per sample, accounting for 92.41% of the total 777.6M parameters rather than 50%. As stated in Appendix B.9, Top-4 routing alone therefore does not imply half the expert FLOPs or a particular latency reduction.
2. The Actual Source of MCRA's Measured Speedup over Capacity-Matched LoRA and DoRA in Table 7
- As shown in Table 7 and Appendices B.8–B.9, under closely matched trainable parameter budgets of approximately 122M–125M, MCRA requires 1041.9 GFLOPs for forward computation—slightly less than 1057.9 GFLOPs for high-rank LoRA and 1113.4 GFLOPs for DoRA. Meanwhile, MCRA achieves a measured inference latency of ms per sample, which is 1.83× faster than LoRA at ms and 2.17× faster than DoRA at ms.
- This measured latency advantage does not come from halving expert computation. Instead, it stems from the architectural execution differences illustrated in Fig. 1a and Table 13. Under the same parameter budget, LoRA and DoRA must perform low-rank updates or weight decomposition in the Query and Value linear modules across all 33 Transformer layers of ESM2-650M. In contrast, MCRA keeps all 33 layers of the ESM2 backbone fully frozen without inserting any internal operators, performing only a single weighted sum of the hidden states from the last 16 layers and executing adaptation in a parallel side branch.
Response to W4:
We fully respect your perspective. Indeed, cross-layer weighting, grouped mixture attention [1], and sparse mixture-of-experts routing [2, 3] are established basic operators in general computer vision and natural language processing. However, this work is not a simple transfer of these general components to the protein domain. Instead, we propose a joint adaptation architecture to address the coexistence of depth heterogeneity and sequence-context heterogeneity in downstream transfer from frozen protein language models. Through experiments and analyses, we reveal three domain-specific findings:
1. We demonstrate task-dependent preferences for representation depth and context scale in protein transfer, with ablation results supporting their complementary roles.
Previous studies on protein transfer either mainly investigated the attention mechanisms or representation properties of different pretrained layers [4, 5], applied uniform fine-tuning operators to fixed layers [6, 7, 8], or used multi-scale feature fusion with fixed depths [9].
By jointly learning task-level layer fusion in Eqs. 3–4 and residue-level scale routing in Eqs. 6–11, Figs. 2 and 4 show that different protein tasks have distinct preferences along two complementary dimensions: representation depth and local context scale.
As described in Section 4.2, even among structure-related tasks, Contact prefers the earliest selected layer—with layer 18 receiving the highest weight—combined with the Medium convolutional receptive field, while Fold Cls mainly uses early-to-intermediate layers 22–25 together with the XLarge receptive field.
Similarly, among whole-protein property tasks, DeepSol prefers deeper intermediate layers 27–32—peaking at layer 29—and the XLarge receptive field, while Human-PPI relies on intermediate layers 26–32—peaking at layer 26—and the Small receptive field.
The three-seed ablation experiments in Tables 5–6 further confirm that the full model, which retains both learnable layer fusion and residue-level adaptive routing, outperforms the final-layer-only variant w/o Layers on all 13 tasks and surpasses the task-shared routing variant TF-Mix on 11 of 13 tasks.
2. We observe a post-hoc association between residue-level routing and local 3D structural environments using sequence-only inputs.
As described in Appendix A.1 and Tables 8 and 10, MCRA uses only amino acid sequences during training and inference, without 3D structural coordinates, contact maps, DSSP secondary structure labels, or solvent accessibility as auxiliary inputs.
In the single-protein post-hoc structural alignment analysis of TEM-1 -lactamase in Fig. 6, Fig. 10, and Table 19 of Appendix D, residue-level routing shows scale adjustments associated with local structural microenvironments, despite an overall preference for the Large scale.
Ordered -helix and -strand regions retain higher proportions of Medium scale routing than random coil/loop regions, with average probabilities of 0.22 and 0.17 compared with 0.07 for loop regions.
Meanwhile, solvent-buried residues assign a probability of 0.23 to the Medium scale. For intermediate and exposed surface residues, the Large scale probability increases from 0.68 to 0.84 and 0.88, while the Medium probability decreases to 0.09 and 0.08.
As carefully stated in Appendix D, this illustrative case suggests that residue-level routing can capture context-scale cues associated with local structural environments from sequence alone, even without structural supervision.
Response to Q1:
We fully understand and agree with your concern about the statistical variance of single-seed results. Before submission, due to the large computational cost of hundreds of fine-tuning experiments, we were unable to complete repeated runs with multiple random seeds, and we acknowledged this limitation.
During the rebuttal period, we have now completed three-seed evaluations across all 13 tasks for MCRA, MCRA-Small, all 8 PLM baselines, and all ablation variants. We have uploaded the revised PDF with all new results and text updates highlighted in blue in Tables 1–2 and Tables 5–6, as well as in Tables G1–G3 of the General Comment above.
Response to W2:
Thank you very much for your careful consideration of the granularity of our ablation experiments. Based on the quantitative results in the main paper and appendices, we explain the independent contributions of residue-wise routing and learnable layer fusion compared with their coarse-grained alternatives.
1. Independent Evidence for the Benefits of Residue-Wise Routing over Protein-Level Routing
Although Tables 5–6 and Eq. 19 in Appendix B.11 directly examine input dependence through TF-Mix, two quantitative findings in our paper further separate the contribution of residue-level granularity from protein-level input-dependent routing.
Evidence A: Residue-Level Scale Differences Across 3D Structural Microenvironments Within a Single Protein
By mathematical definition, protein-level input-dependent routing can generate only one weight vector for a given protein sequence, forcing all residues in the protein to use the exact same mixture of convolutional kernel scales.
However, in our post-hoc structural alignment analysis of TEM-1 β-lactamase in Section 4.2 and Appendix D, supported by Figs. 6 and 10, Table 19, and Eq. 30, we measured routing probability distributions across different structural regions within the same protein chain:
Protein-level routing cannot provide different context processing for the core and surface, or for helices and loops, within the same protein through its routing weights. Such residue-level adaptation to local microenvironments is especially important for Contact prediction, which directly relies on residue-pair representations in Eq. 14. On this task, MCRA achieves a substantial three-seed improvement of 0.0911 over T-Adapter with the same backbone, raising Top-L/5 precision from 0.7555 to 0.8466—and up to 0.8500 for MCRA-Small—in Tables 1 and 5.