- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces UPLiFT, an efficient activation-space fine-tuning framework for protein language models. Its core motivation is to overcome the structural limitations of previous curvature-tuning (CT) methods, which lack sufficient affine-like control (gain and offset adjustments) under frozen weights. By redesigning the parameterized activation function into an additive form, UPLiFT achieves high parameter efficiency while preserving strong adaptation capabilities. Additionally, a lightweight Bayesian extension introduces a stochastic distribution over the activation parameters, providing built-in uncertainty quantification and improved model calibration.
Critical strengths
- The motivation to overcome the gain/offset limitations of traditional activation tuning (like CT) via an affine-like formulation is intuitive.
- Extending the deterministic activation parameters into a stochastic framework (using Beta distributions) is a neat touch. It provides built-in uncertainty quantification (UQ) and improves calibration metrics (like NLL and ECE) almost for free, without messing with heavy model weights.
Critical weaknesses
- The experimental comparisons feel a bit disconnected from the core motivation. Since the primary claim is that UPLiFT fixes the structural flaws of CT (Curvature Tuning), it would be much more convincing to see direct and comprehensive comparisons against the original CT across all tasks. If the improvements are heavily tied to protein representation spaces, the paper needs more dedicated discussion and motivation regarding why protein models specifically benefit from this.
- While the parameter efficiency is impressive, the performance gains are somewhat marginal compared to standard LoRA. It would be very helpful to see an ablation with LoRA rank 1 (or even rank 4). Comparing UPLiFT against an extremely low-rank LoRA baseline would better evidence whether UPLiFT's success stems from true architectural superiority.
Questions for authors
See weaknesses
4 comments
Author Response to Reviewer ZyY7 (part 2)
(W2) While the parameter efficiency is impressive, the performance gains are somewhat marginal compared to standard LoRA. It would be very helpful to see an ablation with LoRA rank 1 (or even rank 4)”
We would like to clarify that our central claim concerns competitive predictive performance with substantially fewer trainable parameters. We do not claim that the proposed activation-only adaptation method is outperforming LoRA tuning. Following your suggestion, we have added a LoRA rank sweep covering (Table 16) for PTM multi-label classification tasks. Our original choice of follows Zhang et al. (2026), from which we adopted the prediction architecture and training protocol. In these additional experiments, we characterize the trade-off across smaller LoRA budgets without separately re-tuning the remaining hyperparameters for each rank. Specifically, we keep the prediction head network the same while examining the rank sensitivity here.
Rank-1 LoRA achieves F1 , compared with for UPLiFT. But it trains 775,688 parameters versus UPLiFT’s 231,936--around 70% reduction for an absolute F1 difference of . Lower-rank LoRA can improve performance relative to the original rank-16 configuration, although performance does not vary monotonically with rank. For example, rank-4 LoRA reaches F1 with 1,121,288 trainable parameters. These results extend the comparison beyond rank 16 and clarify our claim that UPLiFT offers competitive prediction quality while adapting no model weights at a substantially smaller parameter budget. These rank sweep results are added in Appendix H.
Furthermore, the practical benefit is supported by our new compute measurements (requested by other reviewers) with PTM as a representative task. In comparison to rank-16 LoRA, deterministic UPLiFT reduces peak allocated GPU memory from 22.38 to 18.14 GB and per-epoch training time from 495.11 to 481.85 seconds. Note that, with , the training time for LoRA can be lowered, but the GPU memory stays around the same (22.27 GB). Furthermore, such significant contrast in GPU usage between LoRA and UPLiFT persists for ESMC-600M, i.e., 54.79GB (LoRA) and 44.79GB (UPLiFT). Lower memory use, specifically, can matter when choosing hardware with a tighter memory limit. We report these measurements in Appendix P.
Thank you for the rank sweep. I have one remaining concern: the comparison between LoRA and UPLiFT does not control for the prediction head.
Per Sec. 3.1, LoRA trains the full head (~660K parameters), while UPLiFT freezes the head at random initialization and trains only 1,536 activation parameters. In Table 16, the backbone alone has 115,200 trainable parameters for rank-1 LoRA, versus 230,400 for UPLiFT. Rank-1 LoRA is also higher on all four PTM metrics (e.g., F1 0.4148 vs. 0.4080).
So the reported "70% fewer parameters" and "3–11× fewer" come mainly from freezing the head, not from the activation space. Reviewers cp4K and LUY8 raised the same issue, and I could not find a direct answer to it in the responses.
Could you report rank-1 and rank-4 LoRA with the same frozen random head, and list backbone and head parameters as separate columns in the main table and abstract? Without this, I cannot tell whether the benefit comes from activation-space adaptation or from the head setting.
Thank you for suggesting LoRA with a frozen random prediction head. We agree that this comparison helps assess how much adaptation in the PLM alone can drive predictive performance.
While we acknowledge that UPLiFT’s total-parameter advantage over rank-1 LoRA primarily comes from freezing the prediction-head weights, training these weights plays an important role in LoRA’s predictive performance, as the following experiments demonstrate. Moreover, rank-4 and rank-16 LoRA train twice and eight times as many PLM parameters as UPLiFT, respectively.
As requested, we evaluated rank-1 and rank-4 LoRA with the same frozen random prediction-head weights:
Task: PTM classification
| Fine-tuning method | F1 (↑) | MCC (↑) | AUROC (↑) | AUPRC (↑) |
|---|---|---|---|---|
| LoRA (r=1), frozen head | 0.3298 ± 0.0072 | 0.3519 ± 0.0072 | 0.9680 ± 0.0010 | 0.3453 ± 0.0017 |
| LoRA (r=4), frozen head | 0.3500 ± 0.0047 | 0.3725 ± 0.0057 | 0.9659 ± 0.0040 | 0.3604 ± 0.0016 |
| UPLiFT | 0.4080 ± 0.0026 | 0.4149 ± 0.0019 | 0.9796 ± 0.0005 | 0.3787 ± 0.0030 |
All prediction-head weights remain frozen; UPLiFT additionally learns the head’s activation parameters. With frozen head weights, LoRA’s mean F1 is lower than its corresponding trained-head result: 0.3298 versus 0.4148 at rank 1, and 0.3500 versus 0.4217 at rank 4. Both frozen-head configurations nevertheless outperform Head tuning in F1, MCC, and AUPRC, supporting the importance of PLM adaptation. However, PLM-only LoRA adaptation does not recover the performance achieved when the prediction-head weights are also trained.
Crucially, UPLiFT achieves higher mean performance across all four metrics than both frozen-head LoRA configurations, including rank 4 with its larger backbone budget. These results directly address the concern: freezing the prediction-head weights alone does not reproduce UPLiFT’s performance. Its activation parameterization enables effective adaptation of both PLM and head without updating their weights.
We will report backbone and head parameter counts separately (stated below) in the main comparison table (table 2) and clarify that the abstract’s parameter-reduction ratios refer to the total trainable budget:
Number of parameters
- LoRA-tuning (rank-16): 1,843,200 (PLM) / [656,897, 661,514] (head)
- Head-tuning: 0 (PLM) / [656,897, 661,514] (head)
- UPLiFT: 230,400 (PLM) / 1,536 (head)
Author Response to Reviewer ZyY7 (part 1)
We thank the reviewer for their constructive feedback on our work. The following point-by-point responses address the comments, and the related changes are highlighted in purple in the revised PDF. If any clarification is needed, please let us know.
We respectfully disagree that the experiments are disconnected from the motivation. Beyond the toy example that illustrates the adaptation-capacity difference between CT and UPLiFT, Section 3.3 briefly summarizes the CT and UPLiFT comparisons for the residue-level protein properties, with details provided in Appendix E. Specifically, the submitted paper directly compares the two methods on residue-level tasks under both PLM-and-head and head-only activation tuning (Tables 8-9). We have now extended those comparisons to sequence-level classification tasks (Tables 10-11). Beyond these classification tasks, we have also added a regression comparison in the more constrained head-only activation-tuning setting, where we only update the activation of the head (Table 12). Similar to the trend observed for residue-level and sequence-level classification, UPLiFT achieves better performance over CT in this regression task (Pearson correlation: 0.5274 (UPLiFT) vs. 0.0967 (CT)).
UPLiFT vs. CT (Binary localization, Subcellular localization), Table 11
UPLiFT vs. CT (Normal mode vector regression), Table 12
Appendix G of the submitted paper also contains comparison with CT in another setup, where we train prediction head weights with UPLiFT/CT in PLM. We mention this explicitly on pg.6 of the manuscript.
Motivation behind protein tasks
Our evaluation spans eight tasks, including residue- and sequence-level prediction, classification and regression, and diverse biological properties. Such a choice of protein representation space is motivated by the fact that the granularity of the different protein properties, i.e., residue (local) vs. sequence (global), enables us to probe into the effectiveness of our activation-only approach at different levels of fine-tuning task difficulty under a consistent prediction architecture. For example, our results for sequence-level tasks showed that the prediction head’s weights need to be learned (Appendix G), and we cannot achieve LoRA-like adaptation by tuning only the activation space, unlike the residue-level tasks.
To test whether the findings depend on the original backbone, we additionally evaluated ESMC-600M, where UPLiFT remains close to LoRA on PTM (F1 0.4038 vs. 0.4188) and NMV (Pearson 0.681 vs. 0.682), with substantially fewer trainable parameters (Appendix I). The added compute measurements (Appendix P), as requested by other reviewers, also show that the benefit extends beyond parameter counts, specifically in the GPU peak usage during training. While we believe that tasks of broader domains would have demonstrated the generalizability of our approach, the breadth of protein properties of different granularities helped us to systematically evaluate activation-only adaptation under consistent prediction network architectures. We have added a small paragraph in the Discussion and Limitations section to reflect this.