- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates why learning scale parameters separately can improve language-model training even when these parameters can be absorbed into adjacent weight matrices without increasing representational capacity. The authors introduce Learnable Multipliers (LRMs), systematically examining scalar and row/column multipliers at different model locations.
Controlled experiments investigate the relationship between matrix norms, effective scales, and training loss. Placement ablations and long-pretraining experiments demonstrate lower training loss and improved downstream performance across multiple architectures and optimizers.
Critical strengths
The empirical investigation is systematic and practically informative. The authors extensively examine where to introduce scalar and vector multipliers and validate the resulting method across architectures, optimizers, and model sizes. The reductions in pretraining loss and improvements in downstream evaluation support the value of LRM. The discussion of clipping and symmetry-related drift also provides useful implementation guidance.
Critical weaknesses
None
Questions for authors
Appendix E.1 reports symmetry-related drift that can lead to divergence or NaNs, while Appendix E.2 discusses the interaction between multiplier gradients and global clipping. These observations raise a question about stability at scale: as LRM introduces many additional learnable scale parameters, could their collective dynamics introduce additional instability when scaling to substantially larger models and longer training runs? Based on the experiments reported in this paper, have the authors observed any increase in loss spikes, divergence, or sensitivity to multiplier hyperparameters as model size grows? It would be helpful to clarify how far the existing evidence supports the stability of the recommended recipe at larger scales.
2 comments
Thank You
I thank the authors for their response, which has helped me better understand the results. I maintain my score of 3 out of 4. Good luck!
We thank the reviewer for the very positive assessment.
As no weaknesses were pointed out, bellow we address the stability question in detail.
Source of the instability. The only LRM associated instability we observed is related to the symmetry drift: for example (a,b)↦(sa,s−1b) for Q/K multipliers, or a rescaling of the residual stream (App. E.1). As such drift is harmless in exact arithmetic (apart from changing gradients conditioning) but it harmful when reaching the edge of numerical precision range, we observed the following picture: the run was looking fully healthy, eg no spikes, then suddenly diverging to random model loss in a few steps or giving NaN outputs.
We did not observe the negative impact of LRMs on other types of instability, like spikes. In some cases, LRM even helped. In our experiments on classical llama-3 transformer with no QK norm or constant dumpening multiplier on Q or K (like in Falcon-H1) the runs had spikes at moderate learning rates. Same architecture with LRMs initialized at 1 could tolerate much larger learning rates without spikes. The respective experiments were done as a part of joint (width, LR) for μP transfer, and we will include them in the revised version.
Scaling the training duration. This is the main axes for LRM related instability. Our short runs for most of the experiments were under 40 GT of duration, and we did not observe any instability in them. Only for long 240GT pretraining run we first saw symmetry related explosion. The final value of LRM weight decay λlrm=2×10−3 was found by a simple procedure:
The above procedure gave λlrm=2×10−3 with divergences at ≲ 100 GT of training, and hold up until 240 GT.
Scaling the model size. On one hand, we did not observe any further difficulties when scaling model size on axes. For example, the LRM weight decay value λlrm=2×10−3 found on Falcon-H1-0.5B was transferred directly to Llama-3B and its 7B-A1B MoE variant, and we did not observe any instabilities in their 240 GT pretraining runs.
On the other hand, we do not know at the moment how the principled scaling law for LRM weight decay can be derived. The symmetry drift has a peculiar dynamics, and standard μP-like methods seem to not help to analyze the scaling of drift associated instability with model size. To the best of our knowledge, this is an open question.
The scaling of other LRM associated hyperparameters we describe in sec. 3.3: