- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a parameter-efficient fine-tuning method, SCALAR, that learns scalar coefficients over fixed rank-one matrices. It constructs input directions from the leading singular vectors of downstream activations and pairs each direction with multiple fixed random output vectors. The paper motivates this design through a learned-direction reweighting diagnostic and theoretical analyses of activation coverage and initial gradient energy. Experiments on commonsense reasoning, mathematical reasoning, code generation, and MMLU report performance comparable to LoRA with fewer trainable parameters.
Critical strengths
- The experiment covers a wide variety of PEFT methods, including LoRA, LoRA-XS, VeRA, NOLA, and RANDLoRA, which is a plus. However, I have broader concerns regarding the experimental setup, which are discussed below.
Critical weaknesses
- The authors do not report “additional non-trainable parameters” introduced by their method in the main paper and only report trainable parameters, i.e., the frozen activation-derived input vectors () and random output vectors () in Equation (5). This makes the comparison incomplete and difficult to evaluate based solely on trainable parameters.
- The paper's own Table 10 disputes its matched-budget claim on LLaMA3-8B. NOLA beats SCALAR at both matched budgets (0.0028, 0.0057 trainable params %). More broadly, where standard deviations are reported (Tables 3 and 10), most gains between methods fall within them.
- The most obvious baseline, base model, comparison is missing. Although full fine-tuning is included, the authors do not report the base model that was not fine-tuned, making it difficult to assess how much the PEFT methods learned relative to the base model. For MMLU (Table 13), the reported base score from the technical report for Qwen2.5-7B[1] is 74.2, which is within 0.5 points of every FT, LoRA, and SCALAR result. This undermines the claim that SCALAR outperforms LoRA or FT. For math (Table 4), the reported Qwen2.5-7B[1] base scores (GSM8K 85.4, MATH 49.8, with 4-shot ICL) match or exceed all fine-tuned results, including FT. Additionally, the authors should state whether their results are 0-shot or n-shot and compare them to both 0-shot and n-shot baselines. Since training, regardless of parameter count, is more expensive than simply doing n-shot, this will demonstrate the method's effectiveness relative to simple ICL.
- The benchmarks used are potentially contaminated and near saturation to evaluate current models. For some Qwen models, there is evidence of test-set contamination [2]. So, to properly evaluate the effectiveness of SCALAR compared with existing PEFT methods, I would expect evaluation on benchmarks released after the model's release date, so that we can be sure the model’s pre-training was not contaminated by the test set. Some evaluation benchmarks for Qwen 2.5 could be AIME 2025, LiveCodeBench, and SuperGPQA. Or any benchmark that was published after the models release date.
- The method, SCALAR, itself is a minor addition to already known methods. SCALAR combines existing components: activation-SVD input directions (EVA, CorDA), a frozen data-driven input factor (TLoRA), and scalar coefficients over fixed bases (NOLA, VeRA). SVFT [3], which also learns scalars over fixed rank-one matrices , is probably the closest prior method, yet the paper neither properly discusses it nor compares SVFT with SCALAR.
- The theoretical analysis relies on the five assumptions. Assumptions 3.1-3.3 are reasonable for the analysis. However, assumptions 3.4-3.5 are too strong without proper justification. Assumption 3.4 requires . Neither nor is ever estimated. Gradient weighting by can plausibly emphasize activation directions quite different from those that dominate , and the additive bound is not normalized. Without proper justification for assumption 3.4, the “near-optimality” claim in Theorem 3.7 cannot be evaluated. Assumption 3.5 states that LoRA's evolves according to , a flow driven only by the activation covariance that is exactly (normalized) gradient ascent on . Theorem 3.8's conclusion is built from the assumption rather than derived from the task loss, and the paper's own Figure 5 shows that practical LoRA dynamics do not follow this flow: most curves rise sharply early and then decline.
[1]https://arxiv.org/abs/2412.15115
Questions for authors
- Could the authors please share how the models were evaluated? Did they use a standard harness like LM-eval? If not, could the authors please share their rationale for not doing so?
- Also, could the authors please include if they are fine-tuning the base model or fine-tuning instruction tuned-model? Because that could have significant implications on how the models are being evaluated.
- Additionally, more questions are listed in the weakness section above.