- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Randomized Intrinsic Fine-Tuning (RIFT), a parameter-efficient fine-tuning method that adapts pretrained models through sparse linear edits to inter-layer representations. RIFT randomly selects a fixed set of channel interactions and trains only their coefficients, without requiring task-specific support selection or modifying the original model weights. Experiments across commonsense reasoning, language understanding, and mathematical reasoning tasks show competitive performance while training an extremely small fraction of the backbone parameters. Additional ablations investigate the effects of support selection, parameter budget, and intervention placement on adaptation performance.
Critical strengths
-
Important and practical topic: The paper addresses parameter-efficient fine-tuning, an important problem with practical relevance.
-
Extensive experiments: The paper evaluates multiple LLMs across different benchmarks.
Critical weaknesses
-
Limited performance advantages: Although RIFT uses fewer trainable parameters, it does not consistently outperform existing methods. For example, it underperforms SIFT on several benchmarks and shows a substantial performance gap on Llama-3.1-8B GSM8K (49.0% vs. 58.4% for rank-one LoRA). This raises questions about the practical trade-off between parameter efficiency and model performance.
-
Insufficient practical efficiency evaluation: The paper emphasizes parameter efficiency but provides limited evidence of actual computational benefits. The reported GPU memory savings are relatively modest, and there is no systematic comparison of training time, inference latency, or computational overhead.
Questions for authors
See weaknesses.