acceptodds
Under review as a conference paper at ICLR 2027

RIFT: Sparse Representation Fine-Tuning with Random Support

Abstract

Parameter-efficient fine-tuning aims to adapt pretrained models with few trainable parameters, but achieving strong performance under very small budgets remains challenging. Editing inter-layer representations offers a way to reuse pretrained features and adjust downstream computation while preserving the original weights. However, choosing which parameters to train remains a central design decision under such tight budgets. Here we show that randomly selected channel interactions can provide effective adaptation in this space. We introduce Randomized Intrinsic Fine-Tuning (RIFT), which applies a zero-initialized, entry-sparse linear edit to inter-layer representations. RIFT samples a fixed support uniformly from all channel pairs and trains only the selected coefficients. It requires no task-dependent support calibration, and its support can be reconstructed from a seed under a fixed sampling protocol. RIFT can achieve competitive adaptation across commonsense reasoning, language understanding, and mathematical reasoning while training fewer than one hundred-thousandth as many parameters as the backbone, far below the budgets of the evaluated rank-one adapters. Our ablations characterize RIFT's sensitivity to support selection, parameter budget, and intervention placement, and motivate mechanistic studies of how sparse representation edits alter pretrained model behavior.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
PEFT, Representation Fine-Tuning, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NuXhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Randomized Intrinsic Fine-Tuning (RIFT), a parameter-efficient fine-tuning method that adapts pretrained models through sparse linear edits to inter-layer representations. RIFT randomly selects a fixed set of channel interactions and trains only their coefficients, without requiring task-specific support selection or modifying the original model weights. Experiments across commonsense reasoning, language understanding, and mathematical reasoning tasks show competitive performance while training an extremely small fraction of the backbone parameters. Additional ablations investigate the effects of support selection, parameter budget, and intervention placement on adaptation performance.

Critical strengths
  1. Important and practical topic: The paper addresses parameter-efficient fine-tuning, an important problem with practical relevance.

  2. Extensive experiments: The paper evaluates multiple LLMs across different benchmarks.

Critical weaknesses
  1. Limited performance advantages: Although RIFT uses fewer trainable parameters, it does not consistently outperform existing methods. For example, it underperforms SIFT on several benchmarks and shows a substantial performance gap on Llama-3.1-8B GSM8K (49.0% vs. 58.4% for rank-one LoRA). This raises questions about the practical trade-off between parameter efficiency and model performance.

  2. Insufficient practical efficiency evaluation: The paper emphasizes parameter efficiency but provides limited evidence of actual computational benefits. The reported GPU memory savings are relatively modest, and there is no systematic comparison of training time, inference latency, or computational overhead.

Questions for authors

See weaknesses.

Reviewer_m9fjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a parameter-efficient fine-tuning method that works in the activations space. More specifically, while most methods (eg LORA) try to apply low-complexity updates to the model weights, this paper proposes a (condition) activation adaptation method that selectively and sparsely alters the intermediate representations generated throughout the network. The selection mechanism here is random. Similar to some recent PEFT methods, the proposed method focuses on very low dimensional adaptation tasks, where the complexity is below the complexity of rank-1-lora PEFT components (when used throughout the network).

Critical strengths
  • The method is straightforward, relies on simple random selection mechanisms, avoiding complicated schemes.
  • Nicely pushes an alternative avenue for PEFT , by focusing on the activations rather than model parameters directly.
  • Promising results are obtain in certain cases.
Critical weaknesses
  • The results are rather disappointing against the SIFT baseline for smaller models, see Fig 1. SIFT yields high accuracy results at the same complexity.
  • Similarly, GSM8K results (Table 1) are not completely convincing in terms of performance, especially when compared to SIFT baselines. In addition, while going for a smaller
  • Not all comparisons are sensible. Often the baselines contain much higher complexity schemes, which may also cause overfitting. In particular, the comparison against SHiRA caught my eye as SHiRA can deliver much lower dimensional complexities (to the best of my knowledge) than the configurations presented here. SHiRA basically randomly selects weights to be updated -- ie. you may arbitrarily lower its complexity per layer.
  • Evaluation is strictly on certain NLP tasks, though PEFT has many other common applications, including VLM adaptation and generative image model personalization, which are studied in various prior works.
Questions for authors

I don't have any additional questions other than expecting answers to my concerns above.

Reviewer_B1y1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RIFT, a parameter-efficient fine-tuning method that applies sparse linear edits to hidden representations while keeping the pretrained weights frozen. It randomly selects a fixed set of channel interactions and trains only their coefficients, avoiding task-dependent support calibration. Experiments on commonsense reasoning, language understanding, and mathematical reasoning show competitive performance with very few trainable parameters.

Critical strengths
  1. Strong parameter efficiency. RIFT adapts Qwen2.5-7B and Llama-3.1-8B using only 28.6K and 32.8K trainable parameters, respectively, significantly fewer than the evaluated rank-one adapters. On Qwen2.5-7B, it remains within 0.6% of LoRA across multiple tasks while using approximately one-eleventh as many adapter parameters.

  2. Competitive adaptation without task-dependent support calibration. RIFT achieves commonsense reasoning performance close to gradient-selected SIFT on Qwen2.5-7B and Llama-3.1-8B at approximately matched parameter budgets while eliminating the preliminary calibration procedure. It provides a simple alternative for adaptation with very few trainable parameters.

Critical weaknesses
  1. The experiments do not sufficiently isolate the advantages of the proposed sparse representation design.

    (a) Although SIFT provides an approximately parameter-matched comparison in weight space, the evaluated rank-one representation adapters use substantially larger budgets. Comparison with rank-one adapters that are restricted to fewer layers would help determine whether RIFT offers a better accuracy-parameter trade-off under the same budget.

    (b) The saliency probe receives a limited learning-rate search (particularly on Llama) and differs from random support in its treatment of diagonal entries. Although the authors acknowledge these limitations, the reported advantage over the evaluated probe does not establish that random selection outperforms a sufficiently tuned selection procedure under matched settings.

  2. The practical efficiency advantage over existing sparse fine-tuning methods remains insufficiently quantified.

    (a) SIFT already achieves comparable or better performance at approximately the same parameter budget. In Table 1, SIFT outperforms RIFT across all three Qwen2.5-7B benchmarks and achieves the same reported average Commonsense8 accuracy on Llama-3.1-8B. RIFT's main procedural advantage over this baseline is therefore the removal of task-dependent support calibration. Although Section 5.3 reports the gradient-buffer cost of SIFT's calibration procedure, the paper does not quantify how much removing this stage reduces total adaptation time.

    (b) The large reduction in trainable parameters relative to rank-one adapters leads to only modest measured training-memory savings. Table 3 reports peak reserved memory of 22.6 GB for RIFT, compared with 23.5 GB for rank-one LoRA and 23.6 GB for SIFT on Qwen2.5-7B. Although the authors acknowledge that these savings are modest, the results establish a small trainable-parameter budget more clearly than a substantial reduction in overall adaptation costs.

Questions for authors

Please refer to the weaknesses discussed above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.