UPLiFT: Uncertainty Parameterized Activation for Lightweight Fine-Tuning

Learned protein representations contribute to the broad utility of pretrained protein language models (PLMs), often enhanced by task-specific fine-tuning where parameter-efficient techniques are imperative due to large network size. Our work takes an orthogonal approach to such fine-tuning: rather than optimizing the network weights, we formulate adaptation as an optimization task over the parameterized activation space. Motivated by the prior work on curvature-tuning, we propose **UPLiFT**, **U**ncertainty **P**arameterized activation for **Li**ghtweight **F**ine-**T**uning, with enhanced adaptation capacity when the underlying model—whether pretrained or randomly initialized—remains fixed, for significantly improved parameter efficiency. Furthermore, its learnable non-linearity naturally allows to model uncertainty over the activation space through fine-tuning in a variational inference framework. Evaluation across multiple protein datasets shows that UPLiFT performs competitively while requiring $3-11\times$ fewer trainable parameters than weight-only adaptations. Bayesian UPLiFT further improves calibration while largely preserving the predictive performance of deterministically fine-tuned models. The narrow predictive performance margin with the low-rank adaptation baseline in residue-level tasks highlights the potential of our proposed approach as a parameter-efficient fine-tuning technique for PLM-based predictors.