acceptodds
Under review as a conference paper at ICLR 2027

UPLiFT: Uncertainty Parameterized Activation for Lightweight Fine-Tuning

Abstract

Learned protein representations contribute to the broad utility of pretrained protein language models (PLMs), often enhanced by task-specific fine-tuning where parameter-efficient techniques are imperative due to large network size. Our work takes an orthogonal approach to such fine-tuning: rather than optimizing the network weights, we formulate adaptation as an optimization task over the parameterized activation space. Motivated by the prior work on curvature-tuning, we propose **UPLiFT**, **U**ncertainty **P**arameterized activation for **Li**ghtweight **F**ine-**T**uning, with enhanced adaptation capacity when the underlying model—whether pretrained or randomly initialized—remains fixed, for significantly improved parameter efficiency. Furthermore, its learnable non-linearity naturally allows to model uncertainty over the activation space through fine-tuning in a variational inference framework. Evaluation across multiple protein datasets shows that UPLiFT performs competitively while requiring fewer trainable parameters than weight-only adaptations. Bayesian UPLiFT further improves calibration while largely preserving the predictive performance of deterministically fine-tuned models. The narrow predictive performance margin with the low-rank adaptation baseline in residue-level tasks highlights the potential of our proposed approach as a parameter-efficient fine-tuning technique for PLM-based predictors.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Parameter-efficient fine-tuning, Activation space tuning, Uncertainty quantification, Bayesian learning, Protein language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZyY7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UPLiFT, an efficient activation-space fine-tuning framework for protein language models. Its core motivation is to overcome the structural limitations of previous curvature-tuning (CT) methods, which lack sufficient affine-like control (gain and offset adjustments) under frozen weights. By redesigning the parameterized activation function into an additive form, UPLiFT achieves high parameter efficiency while preserving strong adaptation capabilities. Additionally, a lightweight Bayesian extension introduces a stochastic distribution over the activation parameters, providing built-in uncertainty quantification and improved model calibration.

Critical strengths
  1. The motivation to overcome the gain/offset limitations of traditional activation tuning (like CT) via an affine-like formulation is intuitive.
  2. Extending the deterministic activation parameters into a stochastic framework (using Beta distributions) is a neat touch. It provides built-in uncertainty quantification (UQ) and improves calibration metrics (like NLL and ECE) almost for free, without messing with heavy model weights.
Critical weaknesses
  1. The experimental comparisons feel a bit disconnected from the core motivation. Since the primary claim is that UPLiFT fixes the structural flaws of CT (Curvature Tuning), it would be much more convincing to see direct and comprehensive comparisons against the original CT across all tasks. If the improvements are heavily tied to protein representation spaces, the paper needs more dedicated discussion and motivation regarding why protein models specifically benefit from this.
  2. While the parameter efficiency is impressive, the performance gains are somewhat marginal compared to standard LoRA. It would be very helpful to see an ablation with LoRA rank 1 (or even rank 4). Comparing UPLiFT against an extremely low-rank LoRA baseline would better evidence whether UPLiFT's success stems from true architectural superiority.
Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer ZyY7 (part 1)

We thank the reviewer for their constructive feedback on our work. The following point-by-point responses address the comments, and the related changes are highlighted in purple in the revised PDF. If any clarification is needed, please let us know.

(W1) “The experimental comparisons feel a bit disconnected from the core motivation.”

We respectfully disagree that the experiments are disconnected from the motivation. Beyond the toy example that illustrates the adaptation-capacity difference between CT and UPLiFT, Section 3.3 briefly summarizes the CT and UPLiFT comparisons for the residue-level protein properties, with details provided in Appendix E. Specifically, the submitted paper directly compares the two methods on residue-level tasks under both PLM-and-head and head-only activation tuning (Tables 8-9). We have now extended those comparisons to sequence-level classification tasks (Tables 10-11). Beyond these classification tasks, we have also added a regression comparison in the more constrained head-only activation-tuning setting, where we only update the activation of the head (Table 12). Similar to the trend observed for residue-level and sequence-level classification, UPLiFT achieves better performance over CT in this regression task (Pearson correlation: 0.52740.5274 (UPLiFT) vs. 0.09670.0967 (CT)).

UPLiFT vs. CT (Binary localization, Subcellular localization), Table 11

TaskMethodsAccuracy (↑)F1 (↑)AUROC (↑)AUPRC (↑)
Binary Localizationdet. UPLiFT0.8772 ± 0.01630.8946 ± 0.01550.9373 ± 0.01050.9441 ± 0.0107
CT0.6549 ± 0.03830.7697 ± 0.01920.8023 ± 0.03140.7986 ± 0.0493
Subcellular Localizationdet. UPLiFT0.6311 ± 0.02110.4551 ± 0.02120.8760 ± 0.01340.5103 ± 0.0121
CT0.2855 ± 0.00910.0572 ± 0.01080.6617 ± 0.01250.2297 ± 0.0033

UPLiFT vs. CT (Normal mode vector regression), Table 12

MethodPearson (↑)MAE (↓)RMSE (↓)
det. UPLiFT0.5274 ± 0.00720.0488 ± 0.00070.0671 ± 0.0006
CT0.0967 ± 0.04380.1314 ± 0.00910.1707 ± 0.0117

Appendix G of the submitted paper also contains comparison with CT in another setup, where we train prediction head weights with UPLiFT/CT in PLM. We mention this explicitly on pg.6 of the manuscript.

Motivation behind protein tasks

Our evaluation spans eight tasks, including residue- and sequence-level prediction, classification and regression, and diverse biological properties. Such a choice of protein representation space is motivated by the fact that the granularity of the different protein properties, i.e., residue (local) vs. sequence (global), enables us to probe into the effectiveness of our activation-only approach at different levels of fine-tuning task difficulty under a consistent prediction architecture. For example, our results for sequence-level tasks showed that the prediction head’s weights need to be learned (Appendix G), and we cannot achieve LoRA-like adaptation by tuning only the activation space, unlike the residue-level tasks.

To test whether the findings depend on the original backbone, we additionally evaluated ESMC-600M, where UPLiFT remains close to LoRA on PTM (F1 0.4038 vs. 0.4188) and NMV (Pearson 0.681 vs. 0.682), with substantially fewer trainable parameters (Appendix I). The added compute measurements (Appendix P), as requested by other reviewers, also show that the benefit extends beyond parameter counts, specifically in the GPU peak usage during training. While we believe that tasks of broader domains would have demonstrated the generalizability of our approach, the breadth of protein properties of different granularities helped us to systematically evaluate activation-only adaptation under consistent prediction network architectures. We have added a small paragraph in the Discussion and Limitations section to reflect this.

AuthorsOpenReview ↗

Author Response to Reviewer ZyY7 (part 2)

(W2) While the parameter efficiency is impressive, the performance gains are somewhat marginal compared to standard LoRA. It would be very helpful to see an ablation with LoRA rank 1 (or even rank 4)”

We would like to clarify that our central claim concerns competitive predictive performance with substantially fewer trainable parameters. We do not claim that the proposed activation-only adaptation method is outperforming LoRA tuning. Following your suggestion, we have added a LoRA rank sweep covering r∈{1,4,8,16}r\in \lbrace{1,4,8,16\rbrace} (Table 16) for PTM multi-label classification tasks. Our original choice of r=16r=16 follows Zhang et al. (2026), from which we adopted the prediction architecture and training protocol. In these additional experiments, we characterize the trade-off across smaller LoRA budgets without separately re-tuning the remaining hyperparameters for each rank. Specifically, we keep the prediction head network the same while examining the rank sensitivity here.

Rank-1 LoRA achieves F1 0.41480.4148, compared with 0.40800.4080 for UPLiFT. But it trains 775,688 parameters versus UPLiFT’s 231,936--around 70% reduction for an absolute F1 difference of 0.00680.0068. Lower-rank LoRA can improve performance relative to the original rank-16 configuration, although performance does not vary monotonically with rank. For example, rank-4 LoRA reaches 0.42170.4217 F1 with 1,121,288 trainable parameters. These results extend the comparison beyond rank 16 and clarify our claim that UPLiFT offers competitive prediction quality while adapting no model weights at a substantially smaller parameter budget. These rank sweep results are added in Appendix H.

Furthermore, the practical benefit is supported by our new compute measurements (requested by other reviewers) with PTM as a representative task. In comparison to rank-16 LoRA, deterministic UPLiFT reduces peak allocated GPU memory from 22.38 to 18.14 GB and per-epoch training time from 495.11 to 481.85 seconds. Note that, with r=1r=1, the training time for LoRA can be lowered, but the GPU memory stays around the same (22.27 GB). Furthermore, such significant contrast in GPU usage between LoRA and UPLiFT persists for ESMC-600M, i.e., 54.79GB (LoRA) and 44.79GB (UPLiFT). Lower memory use, specifically, can matter when choosing hardware with a tighter memory limit. We report these measurements in Appendix P.

Thank you for the rank sweep. I have one remaining concern: the comparison between LoRA and UPLiFT does not control for the prediction head.

Per Sec. 3.1, LoRA trains the full head (~660K parameters), while UPLiFT freezes the head at random initialization and trains only 1,536 activation parameters. In Table 16, the backbone alone has 115,200 trainable parameters for rank-1 LoRA, versus 230,400 for UPLiFT. Rank-1 LoRA is also higher on all four PTM metrics (e.g., F1 0.4148 vs. 0.4080).

So the reported "70% fewer parameters" and "3–11× fewer" come mainly from freezing the head, not from the activation space. Reviewers cp4K and LUY8 raised the same issue, and I could not find a direct answer to it in the responses.

Could you report rank-1 and rank-4 LoRA with the same frozen random head, and list backbone and head parameters as separate columns in the main table and abstract? Without this, I cannot tell whether the benefit comes from activation-space adaptation or from the head setting.

Thank you for suggesting LoRA with a frozen random prediction head. We agree that this comparison helps assess how much adaptation in the PLM alone can drive predictive performance.

While we acknowledge that UPLiFT’s total-parameter advantage over rank-1 LoRA primarily comes from freezing the prediction-head weights, training these weights plays an important role in LoRA’s predictive performance, as the following experiments demonstrate. Moreover, rank-4 and rank-16 LoRA train twice and eight times as many PLM parameters as UPLiFT, respectively.

As requested, we evaluated rank-1 and rank-4 LoRA with the same frozen random prediction-head weights:

Task: PTM classification

Fine-tuning methodF1 (↑)MCC (↑)AUROC (↑)AUPRC (↑)
LoRA (r=1), frozen head0.3298 ± 0.00720.3519 ± 0.00720.9680 ± 0.00100.3453 ± 0.0017
LoRA (r=4), frozen head0.3500 ± 0.00470.3725 ± 0.00570.9659 ± 0.00400.3604 ± 0.0016
UPLiFT0.4080 ± 0.00260.4149 ± 0.00190.9796 ± 0.00050.3787 ± 0.0030

All prediction-head weights remain frozen; UPLiFT additionally learns the head’s activation parameters. With frozen head weights, LoRA’s mean F1 is lower than its corresponding trained-head result: 0.3298 versus 0.4148 at rank 1, and 0.3500 versus 0.4217 at rank 4. Both frozen-head configurations nevertheless outperform Head tuning in F1, MCC, and AUPRC, supporting the importance of PLM adaptation. However, PLM-only LoRA adaptation does not recover the performance achieved when the prediction-head weights are also trained.

Crucially, UPLiFT achieves higher mean performance across all four metrics than both frozen-head LoRA configurations, including rank 4 with its larger backbone budget. These results directly address the concern: freezing the prediction-head weights alone does not reproduce UPLiFT’s performance. Its activation parameterization enables effective adaptation of both PLM and head without updating their weights.

We will report backbone and head parameter counts separately (stated below) in the main comparison table (table 2) and clarify that the abstract’s parameter-reduction ratios refer to the total trainable budget:

Number of parameters

  • LoRA-tuning (rank-16): 1,843,200 (PLM) / [656,897, 661,514] (head)
  • Head-tuning: 0 (PLM) / [656,897, 661,514] (head)
  • UPLiFT: 230,400 (PLM) / 1,536 (head)
Reviewer_cp4KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UPLiFT adapts a frozen protein language model and a frozen, randomly initialized prediction head by learning activation-function parameters. It extends this approach with Beta-distributed curvature parameters for uncertainty estimation. Experiments use ESM-2 150M across residue- and sequence-level protein tasks. The method trains approximately 0.232M parameters, about 9.3% of the compared LoRA configuration, and approaches LoRA on PTM and NMV. The stochastic variant improves some average calibration metrics, while predictive performance remains below the baselines on several other tasks.

Critical strengths
  1. Adaptation with very few trainable parameters. UPLiFT adapts the frozen model through activation-function parameters, using approximately 0.232M trainable parameters, about 9.3% of the compared LoRA configuration.

  2. Results close to LoRA on selected tasks. UPLiFT obtains PTM F1 of 0.4080 versus LoRA's 0.4089 and NMV Pearson correlation of 0.6738 versus 0.6720. These results suggest that activation-based adaptation is a promising direction, although the benefit is not consistent across tasks.

Critical weaknesses
  1. The parameter-budget/performance trade-off is not sufficiently characterized. Table 2 establishes a much smaller trainable parameter count than the rank-16 LoRA configuration and close results on PTM and NMV. However, the mean results are lower on several other tasks, including all three sequence-level tasks relative to head-only tuning. The authors discuss this limitation, and Appendix G shows that training the prediction head improves sequence-level results. The remaining question is whether UPLiFT offers a better trade-off than lower-budget LoRA configurations, rather than merely using fewer parameters than the selected rank-16 baseline. A small budget sweep with consistent head treatment and separate backbone/head parameter counts would clarify this central comparison. Since the trained head alone has more parameters than the default UPLiFT budget, an exactly matched total of 0.232M should not be required while keeping that same head trainable.

  2. The practical compute cost of the Bayesian extension is not quantified. Table 3 uses 10 prediction samples, and Appendix L reports longer training schedules for several Bayesian experiments. These choices are disclosed, but representative runtime and memory measurements are missing. Such measurements would help users assess the calibration benefit against its computational cost. This does not challenge the demonstrated reduction in trainable parameters or establish that the method is computationally inefficient.

Questions for authors
  1. Could the authors provide a small LoRA budget sweep on representative tasks, alongside the existing UPLiFT variants, with consistent prediction-head treatment and separate backbone/head parameter counts? The head-training ablation in Appendix G is helpful; the remaining goal is to compare the attainable parameter/performance trade-offs.

  2. On a representative task, what are the training time, peak memory, and prediction time for deterministic and Bayesian UPLiFT under their reported training schedules and prediction sample counts? This would help assess the cost of the uncertainty improvement.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer cp4K

We appreciate the reviewer’s effort in providing insightful comments on our work. Below, we addressed each of the weaknesses/questions, and we uploaded the revised PDF accordingly, where the changes are in purple. We believe the responses are directly addressing the comments, but if there is any question/confusion, we are happy to provide any further clarification.

(W1/Q1) “The parameter-budget/performance trade-off is not sufficiently characterized.”

We have added a LoRA rank sweep covering r∈{1,4,8,16}r\in \lbrace{1,4,8,16\rbrace} for the PTM multi-label classification task, with separate backbone and prediction-head parameter counts (Table 16, Appendix H). Our original choice of r=16r=16 follows Zhang et al. (2026), from which we adopted the prediction architecture and training protocol. In these additional experiments, we characterize the trade-off across smaller LoRA budgets without separately re-tuning the remaining hyperparameters for each rank. Specifically, we keep the prediction head network the same while examining the rank sensitivity here.

Lower-rank LoRA can improve performance relative to the original rank-16 configuration, although performance does not vary monotonically with rank. Nevertheless, UPLiFT retains competitive performance at a substantially smaller total budget. Relative to rank-1 LoRA, it trains ~70% fewer parameters, with an absolute F1 difference of 0.0068. These results extend the parameter-performance comparison beyond the originally selected rank-16 baseline.

For consistent “head treatment”, all LoRA configurations use the same fully trainable prediction head. And the UPLiFT result is from the activation-only setup (Table 2), i.e., we learn UPLiFT activation parameters in both the backbone and head while keeping their weights frozen. We also report these parameter counts separately as requested.

Reference:

Jingjie Zhang, et al. Learning the PTM code through a coarse-to-fine mechanism-aware framework. Nature Communications, 2026.

(W2/Q2) “The practical compute cost of the Bayesian extension is not quantified.”

We appreciate the reviewer’s interest in compute cost for uncertainty quantification. We would like to clarify that the strength of the regularization controls how fast we improve predictive performance on the validation set in Bayesian training, and longer training epochs are not always needed, as is the case for catalytic site prediction.

In Appendix P, we have added measurements of training GPU memory, per-epoch training time, and prediction time for deterministic and Bayesian UPLiFT on the PTM classification task, alongside LoRA. The following statistics for ESM2-150M are derived from a single seed of 5 epochs of training (Table 22).

Batch SizeFine-tuning MethodTrain Peak GPU Mem. (GB)Train Time (s)Inference Time (s)
256LoRA (r=16)22.38495.11 ± 0.1943.46 ± 0.21
det. UPLiFT18.14481.85 ± 0.3643.93 ± 0.61
Bayesian UPLiFT18.32528.21 ± 1.3744.65 ± 1.31
1024LoRA (r=16)87.22451.99 ± 0.3940.86 ± 0.12
det. UPLiFT70.92450.37 ± 0.7042.10 ± 0.61
Bayesian UPLiFT71.62465.20 ± 1.3342.50 ± 1.18

On PTM with batch size=256, the Bayesian extension increases per-epoch training time by 9.6%, while the peak allocated GPU memory stays near the same as deterministic UPLiFT. It nevertheless uses 18.1% less GPU memory than LoRA. With similar memory, for M=1M=1, Bayesian UPLiFT has slightly higher measured inference time than LoRA. When Monte Carlo samples are evaluated, inference time increases approximately in proportion to the number of samples (M×inference time (M=1)M\times inference\ time\ (M=1)).

Such a trend of GPU usage and training time by Bayesian UPLiFT is also observed for a larger batch size of 1024. Particularly, the absolute reduction of GPU memory grows proportionally with the batch size. We would like to emphasize that the training of the Bayesian variant does not add significant GPU overhead compared to deterministic UPLiFT.

Dear Reviewer cp4K,

The authors have submitted their rebuttal and we are closing in on the end of the discussion period. Please take some time to carefully read over the rebuttal and respond whether your concerns have been addressed.

Best, AC

Thank you for the additional experiments. The LoRA rank sweep and frozen-head comparisons substantially address my parameter-budget concern. The runtime and memory measurements are also helpful, but single-sample timings do not fully capture the cost of the reported multi-sample Bayesian predictions. The new budget comparisons mainly cover PTM, while performance across the original tasks remains mixed. I maintain my overall rating.

Thank you for acknowledging that the additional rank sweep and frozen-head comparisons substantially address the parameter-budget concern. We would like to clarify your remaining question on computational cost.

The reported M=10M=10 or M=20M=20 samples concern prediction, not training. During Bayesian training, we use one sampled activation configuration per minibatch to estimate the task-loss expectation, while the Beta KL term is evaluated analytically. Training therefore does not require MM forward/backward passes per minibatch and consequently, the sample count (MM) does not affect the training cost of the Bayesian variant.

At inference, our implementation evaluates the samples (Monte Carlo) sequentially, so runtime scales approximately with MM. For example, the measured PTM prediction time at batch size 256 is 44.65 seconds for M=1M=1, giving an estimated M=10M=10 prediction time of approximately 446.5 seconds for the same evaluation pass. We distinguish this extrapolation from a directly measured multi-sample runtime.

PTM was selected in direct response to the original request for measurements on a representative task, with results provided at two batch sizes. These measurements also show that deterministic UPLiFT uses less peak training memory and is faster per epoch than rank-16 LoRA in the benchmarked settings. It does not incur the Bayesian variant’s multi-sample prediction cost.

Our central contribution is activation-only, parameter-efficient adaptation, with uncertainty estimation available as an optional extension. Users can employ deterministic UPLiFT when predictive efficiency is the priority, or Bayesian UPLiFT when uncertainty estimates are required. We do not claim uniform predictive superiority across tasks; our claim concerns the demonstrated performance-parameter trade-off, now supplemented by practical resource measurements.

We respectfully ask that the final assessment reflect the additional evidence addressing the original concerns and supporting UPLiFT’s adaptation contribution. The MM-sample cost applies specifically to Bayesian inference, not to an MM-fold increase in Bayesian training cost, and is not required when using deterministic UPLiFT.

Reviewer_LUY8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UPLiFT, a parameter-efficient fine-tuning method that operates in the activation space. All network weights are frozen, and only a small number of parameters in each neuron's activation function are trained. Building on curvature tuning (CT), the authors modify the form of the activation function and argue that this gives stronger adaptation capacity. They further place a Beta distribution over the activation parameters for uncertainty quantification. Experiments on several protein prediction tasks show that UPLiFT reaches performance close to LoRA on some tasks with far fewer trainable parameters, and the Bayesian variant slightly improves calibration.

Critical strengths
  • The motivation is clear. The paper identifies why CT has limited expressiveness when the weights are frozen, and the proposed modification is simple and direct.
  • Training only activation parameters, without changing any weights, already gets close to LoRA on some tasks.
  • The comparison with CT is fairly thorough, and UPLiFT performs better overall across several settings.
Critical weaknesses
  • Insufficient baselines. The paper lacks PEFT methods with a comparable number of trainable parameters, such as lower-rank LoRA or other methods with fewer parameters, as well as other work that also adapts activation functions. Without these comparisons, it is hard to tell whether the advantage really comes from the activation space.
  • Incomplete efficiency evaluation. Only the number of trainable parameters is compared, and part of the saving comes from freezing the prediction head. The paper does not report common metrics such as memory usage, training time, or inference cost.
  • Limited scope of validation. Only one relatively small backbone model is used, and evaluation is limited to this specific domain.
  • The conclusions are somewhat optimistic. On some tasks, UPLiFT appears close to LoRA under the metrics chosen for the main table, but other metrics in the appendix show a noticeably larger gap.
Questions for authors
  • Does the calibration advantage of the Bayesian variant still hold under the same training budget? How does it compare with temperature scaling and ensemble methods?
  • My other questions are reflected in the Weaknesses section above.
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer LUY8 (part 1)

We thank the reviewer for their constructive comments and questions. We provide a point-by-point response below, and we have highlighted related changes in the revised manuscript in purple. We are happy to provide any further clarification to previous or new comments.

(W1) “Insufficient baselines. The paper lacks PEFT methods with a comparable number of trainable parameters, such as lower-rank LoRA or other methods with fewer parameters, as well as other work that also adapts activation functions. Without these comparisons, it is hard to tell whether the advantage really comes from the activation space.”

We would like to clarify that the main results of UPLiFT (Table 2) are from the activation-only adaptation setting, where we only optimize the parameterized activation function, and all other weights (including the randomly initialized prediction head) are frozen. The empirical results show that this approach is able to learn and be competitive with existing approaches that fine-tune/learn a much larger number of weight parameters, demonstrating the utility of direct modification of the activation space by UPLiFT since none of the weights are altered. That being said, we appreciate the comment on lower-rank LoRA comparison as it can reflect on the activation space’s reach within the LoRA’s wider choice of architectures. We have added the new ablation study as detailed below.

Ablation with LoRA rank

To directly address the request for lower-rank LoRA comparisons, we have added a rank sweep covering r∈{1,4,8,16}r\in \lbrace{1,4,8,16\rbrace} (Table 16, Appendix H) for the PTM multi-label classification task. Our original choice of r=16r=16 follows Zhang et al. (2026), from which we adopted the prediction architecture and training protocol. Note that we keep the prediction head network the same while examining the rank sensitivity here.

Across these ranks, UPLiFT maintains competitive predictive performance with a substantially smaller total trainable parameter budget. Even LoRA with r=1r=1 trains 775,688 parameters, which is approximately 3.3 times as many, when compared with 231,936 for UPLiFT. Its F1 is 0.41480.4148, whereas it is 0.40800.4080 for UPLiFT. Across the complete sweep, LoRA’s mean F1 ranges from 0.40890.4089 to 0.42170.4217, and more importantly, performance does not vary monotonically with rank. Thus, reducing rank changes LoRA’s performance-parameter trade-off, but all tested configurations still require substantially more trainable parameters than UPLiFT.

Comparison for activation adaptation

Regarding comparisons with methods that adapt activation functions, we would like to clarify that the submitted paper already includes extensive comparisons with the original CT in Tables 8 and 9. Specifically, these cover four residue-level tasks under both PLM-plus-head activation tuning and head-only activation tuning. We have further extended these comparisons to sequence-level classification in Tables 10 and 11, and a residue-level regression task in Table 12. All these comparisons directly assess the adaptation capacity of the two activation parameterizations, i.e., UPLIFT vs CT, under frozen network weights.

UPLiFT vs. CT (Binary localization, Subcellular localization), Table 11

TaskMethodsAccuracy (↑)F1 (↑)AUROC (↑)AUPRC (↑)
Binary Localizationdet. UPLiFT0.8772 ± 0.01630.8946 ± 0.01550.9373 ± 0.01050.9441 ± 0.0107
CT0.6549 ± 0.03830.7697 ± 0.01920.8023 ± 0.03140.7986 ± 0.0493
Subcellular Localizationdet. UPLiFT0.6311 ± 0.02110.4551 ± 0.02120.8760 ± 0.01340.5103 ± 0.0121
CT0.2855 ± 0.00910.0572 ± 0.01080.6617 ± 0.01250.2297 ± 0.0033

UPLiFT vs. CT (Normal mode vector regression), Table 12

MethodPearson (↑)MAE (↓)RMSE (↓)
det. UPLiFT0.5274 ± 0.00720.0488 ± 0.00070.0671 ± 0.0006
CT0.0967 ± 0.04380.1314 ± 0.00910.1707 ± 0.0117

This new set of experiments strengthens our previous results (Tables 8-9). Specifically, the distinction is particularly clear when only prediction-head activations are optimized: on binary localization, UPLiFT achieves accuracy 0.87720.8772, compared with 0.65490.6549 for CT; on subcellular localization, the corresponding values are 0.63110.6311 and 0.28550.2855, respectively (Table 11). This advantage also extends to NMV regression, where UPLiFT achieves Pearson correlation of 0.5274 versus 0.0967 for CT, with lower MAE (0.0488 versus 0.1314; Table 12).

AuthorsOpenReview ↗

Author Response to Reviewer LUY8 (part 2)

These results provide direct evidence that our proposed activation parameterization enables substantially stronger adaptation than CT, complementing the comparisons with weight-space fine-tuning. Furthermore, Appendix G of the submitted paper contains comparison with CT in another setup, where we train prediction head weights with UPLiFT/CT in PLM. In this scenario, UPLiFT also outperforms CT. We mention this explicitly on pg.6 of the manuscript.

Reference:

Jingjie Zhang, et al. Learning the PTM code through a coarse-to-fine mechanism-aware framework. Nature Communications, 2026.

(W2) “Incomplete efficiency evaluation....”

We have expanded the efficiency evaluation to include training GPU memory, per-epoch training time, and inference time for PTM multi-label classification as a representative task with ESM2-150M and ESMC-600M.

PLMFine-tuning MethodTrain Peak GPU Mem. (GB)
ESM2-150M (batch size = 256)LoRA (r=16)22.38
det. UPLiFT18.14
ESM2-150M (batch size = 1024)LoRA (r=16)87.22
det. UPLiFT70.92
ESMC-600M (batch size = 256)LoRA (r=16)54.79
det. UPLiFT44.79

For ESM2-150M with batch size=256, the current UPLiFT implementation reduces peak allocated GPU memory from 22.38 to 18.14 GB (18.9% lower) and per-epoch training time from 495.11 to 481.85 seconds (2.7% lower) relative to LoRA. Note, with r=1r=1, the training time for LoRA can be lowered, but the GPU memory stays around the same (22.27 GB). The Bayesian variant takes longer to train, but adds a small overhead to the GPU memory. With larger batch size (1024), the reduction in GPU usage by deterministic UPLiFT is around 16 GB (18.7% lower).

Furthermore, such significant contrast in GPU usage between LoRA and UPLiFT persists for ESMC-600M, i.e., 54.79 GB (LoRA) and 44.79 GB (UPLiFT). These results highlight UPLiFT’s memory efficiency, particularly under a limited GPU memory budget.

We believe these new compute efficiency measurements (added in Table 22 - Appendix P) complement the parameter-count comparison. Specifically, they demonstrate reductions in memory usage (and training time in the deterministic variant) by UPLiFT in the benchmarked settings, while achieving competitive predictive performance.

(W3) “Only one relatively small backbone model is used, and evaluation is limited to this specific domain.”

To address the concern about reliance on one relatively small backbone, we additionally evaluate UPLiFT using ESMC-600M, a different backbone with four times the parameter count of ESM2-150M. The results are included in Appendix I.

For PTM multi-label classification (Table 17), UPLiFT achieves an F1 of 0.4038±0.00270.4038\pm 0.0027, compared with 0.4188±0.00360.4188\pm 0.0036 for LoRA and 0.3156±0.00160.3156\pm 0.0016 for Head tuning. Across all four metrics, UPLiFT achieves higher mean performance than Head tuning and remains competitive with LoRA, while using only 333,312 trainable parameters compared to LoRA’s 5,166,088 (approximately 6.5%). For NMV regression (Table 18), UPLiFT achieves Pearson correlation comparable to LoRA and lower mean MAE and RMSE, while using only 6.5% of LoRA’s trainable parameters. It also outperforms Head tuning across all metrics with substantially fewer trainable parameters. Note that we have kept the hidden neuron dimension in the prediction head the same for this larger model, resulting in the same number of learnable parameters (1536) in the prediction head as it was with the ESM2-150M backbone.

This trend for ESMC-600M is consistent with our previous observations with ESM2-150M. These additional results demonstrate that the favorable performance-parameter trade-off extends beyond the original (relatively smaller) backbone and model scale.

Task: PTM multi-label classification

Method# Params (PLM/Head)F1 (↑)MCC (↑)AUROC (↑)AUPRC (↑)
LoRA tuning3,981,312/1,184,7760.4188±0.00360.4253±0.00340.9818±0.00080.3974±0.0029
Head tuning0/1,184,7760.3156±0.00160.3140±0.00130.9742±0.00040.2747±0.0014
UPLiFT331,776/1,5360.4038±0.00270.4095±0.00510.9790±0.00150.3799±0.0020

Task: Normal mode vector (NMV) regression

Method# Params (PLM/Head)Pearson (↑)MAE (↓)RMSE (↓)
LoRA tuning3,981,312/1,181,1850.682±0.0020.0387±0.00080.0572±0.0006
Head tuning0/1,181,1850.610±0.0030.0436±0.00040.0622±0.0003
UPLiFT331,776/1,5360.681±0.0030.0384±0.00030.0567±0.0003
AuthorsOpenReview ↗

Author Response to Reviewer LUY8 (part 3)

Breadth of evaluation

We would like to clarify the breadth and intended scope of our evaluation. The submitted paper evaluates UPLiFT across eight protein property prediction tasks, covering both residue- and sequence-level prediction, classification and regression, and diverse biological properties. Comparisons with LoRA tuning, Head tuning, and original CT, together with additional ablations, examine adaptation under multiple settings. Thus, the existing validation already covers substantial task diversity within the context of protein property prediction.

Furthermore, such a choice of protein domain is motivated by the fact that the granularity of the different protein properties, i.e., residue (local) vs. sequence (global), enables us to probe into the effectiveness of our activation-only approach at different levels of fine-tuning task difficulty. For example, our results for sequence-level tasks showed that the prediction head’s weights need to be learned for better sequence-level protein property prediction (Appendix G), and we cannot achieve LoRA-like adaptation by tuning only the activation space, unlike the residue-level tasks. Although evaluation in other domains would broaden the evidence for generality, our empirical claims concern protein property prediction, where the evaluation spans diverse tasks and prediction settings. We, therefore, believe the characterization of the validation as limited does not fully reflect its breadth, particularly considering the additional ESMC-600M results.

(W4) “The conclusions are somewhat optimistic On some tasks, UPLiFT appears close to LoRA under the metrics chosen for the main table, but other metrics in the appendix show a noticeably larger gap.”

We would like to emphasize that our central claim concerns competitive predictive performance with substantially fewer trainable parameters. In the main ESM2 comparison, UPLiFT uses only 9.3% of LoRA’s trainable parameters. For example, its PTM F1 is 0.4080 versus 0.4089 for LoRA. Across residue-level tasks, UPLiFT generally improves upon Head tuning while remaining competitive with LoRA.

Additional ESMC-600M experiments (PTM classification, NMV regression; in response to W3; Appendix I) further support this trade-off. Particularly for NMV, UPLiFT achieves comparable Pearson correlation to LoRA and lower mean MAE /RMSE, using only 6.5% of its trainable parameters. These results demonstrate that the benefits extend to a different, substantially larger backbone. The addition of compute efficiency measurement (in response to W2; Appendix P) highlights another practical benefit, e.g., reduction of GPU usage, which can be useful under strict memory constraints without needing to reduce the batch size.

The primary metrics reported in the main table follow the evaluation conventions of the source benchmark studies (described in Appendix B). We reported the additional metrics in the appendix to provide a more comprehensive assessment. We acknowledge that the gap from LoRA is larger on some of these metrics, and our submitted paper is consistent with it. Specifically, we already have the following statement in the main text:

“Although UPLiFT provides competitive performance relative to weight-space fine-tuning baselines, it generally performs below LoRA across both residue- and sequence-level tasks.” (second paragraph in Discussion and Limitations)

Nonetheless, our claim of competitive performance should be understood in the context of the substantial reduction in trainable parameters. Particularly, the strong performance with residue-level tasks, complemented by LoRA-like performance for sequence-level tasks when we learn the weights of the prediction head (Appendix G) with UPLiFT in PLM, provides evidence of activation adaptation as a practical alternative across protein prediction tasks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Reviewers,

We appreciate your engagement during the discussion period. It was highly encouraging for us to see that you found our proposed parameterized activation function UPLiFT interesting, and we believe the revision provides further evidence of UPLiFT's competitive edges over baselines.

We propose UPLiFT, a parameter-efficient fine-tuning method that achieves model adaptation through activation-space optimization while keeping all network weights frozen. Through extensive evaluation across multiple protein property tasks (both residue and sequence level property), we show UPLiFT can get the competitive performance while using much less number of optimization parameters than approaches that fine-tune in weight space, e.g., LoRA in PLM with prediction head; as a result, our benchmarks demonstrate reduced GPU memory usage during training. Moreover, UPLiFT’s formulation offers an optional way to model uncertainty in activation space without introducing any additional parameters.

During the rebuttal we directly examined the concern regarding random prediction-head, following the requests of reviewers for different LoRA ranks. Furthermore, our new results include experiments with ESMC-600M, where the UPLiFT's parameter-performance tradeoff is extended to a larger PLM backbone beyond ESM2-150M from our initial paper. Finally, the revision has also added compute efficiency evaluation of UPLiFT, where it demonstrated moderate GPU memory usage saving (~19% reduction from LoRA).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.