acceptodds
Withdrawn from ICLR 2027

Direction or Magnitude? Learning Scalars over Task-Calibrated Directions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting pretrained models, yet standard low-rank methods still optimize matrix-valued factors whose parameter, optimizer-state, and memory costs grow with model width and adapter rank. Fixed-basis approaches further compress task-specific parameters, but purely random directions may be poorly aligned with downstream data, while storing and combining large collections of fixed bases can still incur substantial memory and computational overhead. We propose scalar, a scalar-only adaptation method that combines activation-derived input directions with fixed random output atoms and optimizes only layer-specific coefficients. The activation-calibrated subspace provides task-relevant input directions, whereas grouped random atoms form a compact output dictionary that can additionally be shared across compatible layers. Experiments on commonsense reasoning, mathematical reasoning, code generation, and general instruction tuning show that \name achieves performance comparable to LoRA with up to 55 fewer trainable parameters. In representative 7B-model experiments, \name reduces peak training memory by 20% and improves training throughput by 6% relative to LoRA, while yielding larger system-efficiency gains over existing random-basis methods. These results show that scalar-only adaptation can improve not only nominal parameter efficiency, but also practical memory consumption and training speed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, fine-tuning, PEFT

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FhSXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a parameter-efficient fine-tuning method, SCALAR, that learns scalar coefficients over fixed rank-one matrices. It constructs input directions from the leading singular vectors of downstream activations and pairs each direction with multiple fixed random output vectors. The paper motivates this design through a learned-direction reweighting diagnostic and theoretical analyses of activation coverage and initial gradient energy. Experiments on commonsense reasoning, mathematical reasoning, code generation, and MMLU report performance comparable to LoRA with fewer trainable parameters.

Critical strengths
  • The experiment covers a wide variety of PEFT methods, including LoRA, LoRA-XS, VeRA, NOLA, and RANDLoRA, which is a plus. However, I have broader concerns regarding the experimental setup, which are discussed below.
Critical weaknesses
  • The authors do not report “additional non-trainable parameters” introduced by their method in the main paper and only report trainable parameters, i.e., the frozen activation-derived input vectors (ara_r) and random output vectors (br,hb_{r,h}) in Equation (5). This makes the comparison incomplete and difficult to evaluate based solely on trainable parameters.
  • The paper's own Table 10 disputes its matched-budget claim on LLaMA3-8B. NOLA beats SCALAR at both matched budgets (0.0028, 0.0057 trainable params %). More broadly, where standard deviations are reported (Tables 3 and 10), most gains between methods fall within them.
  • The most obvious baseline, base model, comparison is missing. Although full fine-tuning is included, the authors do not report the base model that was not fine-tuned, making it difficult to assess how much the PEFT methods learned relative to the base model. For MMLU (Table 13), the reported base score from the technical report for Qwen2.5-7B[1] is 74.2, which is within 0.5 points of every FT, LoRA, and SCALAR result. This undermines the claim that SCALAR outperforms LoRA or FT. For math (Table 4), the reported Qwen2.5-7B[1] base scores (GSM8K 85.4, MATH 49.8, with 4-shot ICL) match or exceed all fine-tuned results, including FT. Additionally, the authors should state whether their results are 0-shot or n-shot and compare them to both 0-shot and n-shot baselines. Since training, regardless of parameter count, is more expensive than simply doing n-shot, this will demonstrate the method's effectiveness relative to simple ICL.
  • The benchmarks used are potentially contaminated and near saturation to evaluate current models. For some Qwen models, there is evidence of test-set contamination [2]. So, to properly evaluate the effectiveness of SCALAR compared with existing PEFT methods, I would expect evaluation on benchmarks released after the model's release date, so that we can be sure the model’s pre-training was not contaminated by the test set. Some evaluation benchmarks for Qwen 2.5 could be AIME 2025, LiveCodeBench, and SuperGPQA. Or any benchmark that was published after the models release date.
  • The method, SCALAR, itself is a minor addition to already known methods. SCALAR combines existing components: activation-SVD input directions (EVA, CorDA), a frozen data-driven input factor (TLoRA), and scalar coefficients over fixed bases (NOLA, VeRA). SVFT [3], which also learns scalars over fixed rank-one matrices uivjTu_i v_j^T, is probably the closest prior method, yet the paper neither properly discusses it nor compares SVFT with SCALAR.
  • The theoretical analysis relies on the five assumptions. Assumptions 3.1-3.3 are reasonable for the analysis. However, assumptions 3.4-3.5 are too strong without proper justification. Assumption 3.4 requires ∥M−cΣ∥2≤ε\|M - c\Sigma\|_2 \le \varepsilon. Neither cc nor ε\varepsilon is ever estimated. Gradient weighting by ∥g∥22\|g\|_2^2 can plausibly emphasize activation directions quite different from those that dominate Σ\Sigma, and the additive 2Rε2R\varepsilon bound is not normalized. Without proper justification for assumption 3.4, the “near-optimality” claim in Theorem 3.7 cannot be evaluated. Assumption 3.5 states that LoRA's AA evolves according to A˙=AΣ−(AΣA⊤)A\dot A = A\Sigma - (A\Sigma A^\top)A, a flow driven only by the activation covariance Σ\Sigma that is exactly (normalized) gradient ascent on Φ\Phi. Theorem 3.8's conclusion is built from the assumption rather than derived from the task loss, and the paper's own Figure 5 shows that practical LoRA dynamics do not follow this flow: most curves rise sharply early and then decline.

[1]https://arxiv.org/abs/2412.15115

[2]https://arxiv.org/abs/2507.10532

[3]https://arxiv.org/abs/2405.19597

Questions for authors
  • Could the authors please share how the models were evaluated? Did they use a standard harness like LM-eval? If not, could the authors please share their rationale for not doing so?
  • Also, could the authors please include if they are fine-tuning the base model or fine-tuning instruction tuned-model? Because that could have significant implications on how the models are being evaluated.
  • Additionally, more questions are listed in the weakness section above.
Reviewer_2gC7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SCALAR, a parameter-efficient fine-tuning method that freezes input directions obtained from an SVD of downstream activations and pairs each input direction with a group of fixed random output atoms. Only layer-specific scalar combination coefficients are optimized. The grouped parameterization separates the coefficient budget from the maximum update rank, while compatible layers can share output dictionaries. The paper motivates the input basis through activation coverage and initial-gradient analysis, and evaluates the method on commonsense reasoning, mathematical reasoning, code generation, and general-domain knowledge evaluation. The reported results show competitive accuracy with fewer trainable parameters and measured training-memory savings.

Critical strengths

The paper reports a consistent improvement over LoRA-XS at matched trainable-parameter budgets across the two main commonsense backbones, providing evidence that the proposed configuration is useful within the fixed-basis adaptation family. It also measures actual peak training memory and throughput rather than relying exclusively on trainable-parameter counts; the representative Qwen2.5-7B setting reports approximately 20% lower peak memory than LoRA. These results make the efficiency claim more practically grounded, although they do not resolve the novelty and robustness concerns below.

Critical weaknesses

W1. Limited methodological novelty beyond existing fixed-basis adaptation methods

The methodological contribution appears incremental relative to existing fixed-basis PEFT approaches. VeRA already freezes and shares random projection matrices while learning layer-specific scaling vectors, and LoRA-XS freezes SVD-derived projections and trains only a small coefficient matrix. In particular, LoRA-XS can itself be expressed as learning scalar coefficients over fixed rank-one atoms; therefore, “scalar-only adaptation” is not, by itself, a substantive distinction. Moreover, activation-SVD input directions are already central to EVA. SCALAR introduces a distinct grouped output dictionary, but its conceptual advance is primarily the combination of established ingredients. Although the paper includes initialization and grouping ablations, these do not fully establish the added value of the proposed dictionary over a simpler activation-calibrated fixed-basis adapter. Consequently, the evidence does not sufficiently demonstrate that the specific parameterization constitutes a substantial methodological advance.

W2. Marginal accuracy gains over strong baselines, with limited evidence of robustness

The accuracy improvements over strong LoRA-based baselines are small and inconsistent across settings. On commonsense reasoning, SCALAR achieves 88.9 versus 88.8 for rank-8 LoRA on Qwen2.5-7B, but 87.3 versus 87.4 on LLaMA3-8B. The comparison with EVA similarly shows only a 0.2-percentage-point improvement on Qwen2.5-7B, without uncertainty estimates in that comparison sufficient to assess the reliability of the difference. These results support competitive accuracy under a smaller trainable-parameter budget, rather than a clear performance advantage. The parameter reduction and reported memory savings are meaningful; however, the representative throughput improvement over LoRA is only approximately 6%, and excludes the one-time calibration cost. Taken together, the results do not convincingly establish a sufficiently substantial and robust practical advantage to compensate for the limited methodological novelty.

Questions for authors
  1. Could you isolate the contribution of the grouped random output dictionary using an activation-SVD input basis held fixed across variants, and compare against both a trainable output factor and a compact fixed-projection coefficient adapter? Please distinguish improvements due to the input basis from those due to the output parameterization.

  2. Could you report per-seed results or uncertainty estimates for the EVA/PiSSA comparisons in Table 12, together with end-to-end training time including calibration for the representative efficiency setting? These would help assess whether the small accuracy differences are reliable and how the measured per-step efficiency translates into total adaptation cost.

Reviewer_AXFROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether PEFT can fix the adaptation directions ahead of time and learn only scalar coefficients on top of them. The authors first run a diagnostic: they take a trained LoRA adapter, freeze its learned update directions, and re-optimize one scalar per direction. On a 7B model this brings the trainable parameters from about 2.5M down to 196 per layer with almost no change in accuracy (Table 2). They then propose SCALAR, which fixes the input directions to the top singular vectors of the input activation covariance, computed on a small calibration set, and pairs each direction with several frozen random output atoms; only the scalar coefficients are trainable. The paper gives a theoretical justification for using activation-derived directions, and evaluates the method on commonsense reasoning, math, and code with Qwen2.5-7B and LLaMA3-8B. SCALAR roughly matches LoRA accuracy while using orders of magnitude fewer trainable parameters, and the system measurements show lower peak memory and higher training throughput.

Critical strengths
  1. The reweighting diagnostic (Table 2, Section 3.1) is clean and somewhat surprising: a trained rank-16 LoRA update collapses to a few hundred scalars per layer with negligible accuracy loss. This is a well-controlled observation that motivates the method directly and is useful on its own.
  2. The efficiency evaluation is more convincing than in most PEFT papers. Instead of stopping at parameter counts, the authors measure peak GPU memory and token throughput on real hardware (Figure 2). The reported savings, roughly 20% memory over LoRA and higher throughput, support the claim that the parameter reduction translates into practical gains.
Critical weaknesses
  1. The main theoretical claim overlaps with existing work. EVA (Paischer et al., 2025) already proves that directions capturing the most activation variance are near-optimal for the initial gradient signal, and proposes using them as a LoRA initialization. The paper cites EVA but never states precisely what Theorem 3.7 adds. As written, the theory section reads as a re-derivation, and the novelty has to be made explicit.
  2. The empirical advantage is mostly about parameter count, not accuracy. In Table 3, SCALAR is on par with or slightly below LoRA and NOLA on both models (e.g., 86.5 vs 86.9 for LoRA r=8 and 87.4 for NOLA on LLaMA3-8B). Since rank-8 LoRA already uses only about 0.26% of parameters, the paper should argue more concretely why the extreme 0.0026% regime matters, ideally with a concrete use case such as storing many adapters or on-device training.
Questions for authors
  1. Please state precisely what the theoretical analysis adds over EVA, and point to a setting where the two analyses lead to different conclusions.
  2. How does SCALAR compare with an EVA-style activation initialization of LoRA at matched parameter budgets, or with A frozen but B trainable? This would separate the benefit of activation directions from the benefit of freezing them.
  3. What is the calibration cost in practice, such as the size of the calibration set and the number of forward passes? How sensitive are the results to the calibration data, and do they change if calibration and training data are strictly disjoint?
  4. The memory and throughput numbers are for Qwen2.5-7B on two H20 GPUs. Would the trend be similar on a single GPU or on larger models, where memory savings might matter more?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.