RIFT: Sparse Representation Fine-Tuning with Random Support

Parameter-efficient fine-tuning aims to adapt pretrained models with few trainable parameters, but achieving strong performance under very small budgets remains challenging. Editing inter-layer representations offers a way to reuse pretrained features and adjust downstream computation while preserving the original weights. However, choosing which parameters to train remains a central design decision under such tight budgets. Here we show that randomly selected channel interactions can provide effective adaptation in this space. We introduce Randomized Intrinsic Fine-Tuning (RIFT), which applies a zero-initialized, entry-sparse linear edit to inter-layer representations. RIFT samples a fixed support uniformly from all channel pairs and trains only the selected coefficients. It requires no task-dependent support calibration, and its support can be reconstructed from a seed under a fixed sampling protocol. RIFT can achieve competitive adaptation across commonsense reasoning, language understanding, and mathematical reasoning while training fewer than one hundred-thousandth as many parameters as the backbone, far below the budgets of the evaluated rank-one adapters. Our ablations characterize RIFT's sensitivity to support selection, parameter budget, and intervention placement, and motivate mechanistic studies of how sparse representation edits alter pretrained model behavior.