Direction or Magnitude? Learning Scalars over Task-Calibrated Directions

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting pretrained models, yet standard low-rank methods still optimize matrix-valued factors whose parameter, optimizer-state, and memory costs grow with model width and adapter rank. Fixed-basis approaches further compress task-specific parameters, but purely random directions may be poorly aligned with downstream data, while storing and combining large collections of fixed bases can still incur substantial memory and computational overhead. We propose scalar, a scalar-only adaptation method that combines activation-derived input directions with fixed random output atoms and optimizes only layer-specific coefficients. The activation-calibrated subspace provides task-relevant input directions, whereas grouped random atoms form a compact output dictionary that can additionally be shared across compatible layers. Experiments on commonsense reasoning, mathematical reasoning, code generation, and general instruction tuning show that \name achieves performance comparable to LoRA with up to 55$\times$ fewer trainable parameters. In representative 7B-model experiments, \name reduces peak training memory by 20% and improves training throughput by 6% relative to LoRA, while yielding larger system-efficiency gains over existing random-basis methods. These results show that scalar-only adaptation can improve not only nominal parameter efficiency, but also practical memory consumption and training speed.