acceptodds
Under review as a conference paper at ICLR 2027

Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers

Abstract

Language-model pretraining typically combines weight decay (WD) on parameter matrices with learnable scales, such as RMSNorm gains. Scales that can be absorbed into adjacent matrices add no representational capacity, yet learning them separately often improves training. We untangle the roles of WD and separately learned scales through a single mechanism: optimizer updates can drive unchecked norm growth, WD restrains this growth but holds matrix norms near potentially suboptimal values, and separately learned scales let the effective matrix scales adapt to the data. We demonstrate this mechanism both empirically, through controlled experiments with Adam and Muon, and theoretically, using the Adam-trained LM head as a case study in which second-moment normalization suppresses the signal from rare informative gradients. We introduce learnable multipliers (LRMs) to extend this scale adaptation systematically across model matrices. LRMs learn diverse feature scales across and within layers while remaining compatible with P learning-rate transfer. Progressively adding scale degrees of freedom across model matrices reduces training loss. In long pretraining across architectures and optimizers, our full LRM configuration delivers substantial downstream gains comparable to the Adam-to-Muon switch.

Primary area:
foundation or frontier models, including LLMs
Keywords:
optimization, LLM pretraining, weight decay, Adam, SDE, reparametrization, maximal update parametrization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YVtmOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates why learning scale parameters separately can improve language-model training even when these parameters can be absorbed into adjacent weight matrices without increasing representational capacity. The authors introduce Learnable Multipliers (LRMs), systematically examining scalar and row/column multipliers at different model locations.

Controlled experiments investigate the relationship between matrix norms, effective scales, and training loss. Placement ablations and long-pretraining experiments demonstrate lower training loss and improved downstream performance across multiple architectures and optimizers.

Critical strengths

The empirical investigation is systematic and practically informative. The authors extensively examine where to introduce scalar and vector multipliers and validate the resulting method across architectures, optimizers, and model sizes. The reductions in pretraining loss and improvements in downstream evaluation support the value of LRM. The discussion of clipping and symmetry-related drift also provides useful implementation guidance.

Critical weaknesses

None

Questions for authors

Appendix E.1 reports symmetry-related drift that can lead to divergence or NaNs, while Appendix E.2 discusses the interaction between multiplier gradients and global clipping. These observations raise a question about stability at scale: as LRM introduces many additional learnable scale parameters, could their collective dynamics introduce additional instability when scaling to substantially larger models and longer training runs? Based on the experiments reported in this paper, have the authors observed any increase in loss spikes, divergence, or sensitivity to multiplier hyperparameters as model size grows? It would be helpful to clarify how far the existing evidence supports the stability of the recommended recipe at larger scales.

2 comments

We thank the reviewer for the very positive assessment.

As no weaknesses were pointed out, bellow we address the stability question in detail.

Source of the instability. The only LRM associated instability we observed is related to the symmetry drift: for example (a,b)↦(sa,s−1b)(a,b)\mapsto(sa,s^{-1}b) for Q/K multipliers, or a rescaling of the residual stream (App. E.1). As such drift is harmless in exact arithmetic (apart from changing gradients conditioning) but it harmful when reaching the edge of numerical precision range, we observed the following picture: the run was looking fully healthy, eg no spikes, then suddenly diverging to random model loss in a few steps or giving NaN outputs.

We did not observe the negative impact of LRMs on other types of instability, like spikes. In some cases, LRM even helped. In our experiments on classical llama-3 transformer with no QK norm or constant dumpening multiplier on Q or K (like in Falcon-H1) the runs had spikes at moderate learning rates. Same architecture with LRMs initialized at 11 could tolerate much larger learning rates without spikes. The respective experiments were done as a part of joint (width, LR) for μ\muP transfer, and we will include them in the revised version.

Scaling the training duration. This is the main axes for LRM related instability. Our short runs for most of the experiments were under 40 GT of duration, and we did not observe any instability in them. Only for long 240GT pretraining run we first saw symmetry related explosion. The final value of LRM weight decay λlrm=2×10−3\lambda_{lrm}=2\times 10^{-3} was found by a simple procedure:

  1. when instability is observed, go back a several checkpoint before instability and relaunch with larger value of λlrm\lambda_{lrm}
  2. If instability is observed again later in the training, repeat 1.
    The above procedure gave λlrm=2×10−3\lambda_{lrm}=2\times 10^{-3} with divergences at ≲\lesssim 100 GT of training, and hold up until 240 GT.

Scaling the model size. On one hand, we did not observe any further difficulties when scaling model size on axes. For example, the LRM weight decay value λlrm=2×10−3\lambda_{lrm}=2\times 10^{-3} found on Falcon-H1-0.5B was transferred directly to Llama-3B and its 7B-A1B MoE variant, and we did not observe any instabilities in their 240 GT pretraining runs.

On the other hand, we do not know at the moment how the principled scaling law for LRM weight decay can be derived. The symmetry drift has a peculiar dynamics, and standard μ\muP-like methods seem to not help to analyze the scaling of drift associated instability with model size. To the best of our knowledge, this is an open question.

The scaling of other LRM associated hyperparameters we describe in sec. 3.3:

  • Initialization scaling should follow mumuP rules for the respective forward multipliers in parametrization that leaves matrix LR and WD constant.
  • Scaling of learning rate follows initialization, so that relative LRM updates stay O(1)O(1). Though we don't provide a rigorous μ\muP derivation for the above scaling, they essentially follow standard mumuP arguments of keeping activations and their updates O(1)O(1), and they also provided expected learning rate transfer (see fig.6 and fig.13). And scaling with depth we left out of the scope of the current paper.
Reviewer_YVtmOpenReview ↗

Thank You

I thank the authors for their response, which has helped me better understand the results. I maintain my score of 3 out of 4. Good luck!

Reviewer_vfrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces learnable multipliers to provide additional flexibility in scaling the matrix layers of language models. The authors argue that weight decay can restrain matrix norm growth, but potentially leaving matrix magnitudes poorly suited to the task at times. By introducing separately learned multipliers, the effective magnitudes can adapt during training. These additional parameters provide optimization flexibility without increasing the representational capacity of the model.

Critical strengths
  • The paper provides a systematic investigation of the interactions among optimizer behavior, matrix norm growth, weight decay, and learned scales. The authors conduct extensive experiments on multiplier placement and examine compatibility with learning rate transfer across model widths.

  • Another strength is the experimental scope, including long training runs across dense, hybrid, and mixture-of-experts models.

Critical weaknesses
  • My main concern is the extent of the paper's novelty. The general idea that learning quantities such as the learning rate can improve training and the resulting network's inference performance is already well known in the community. In this context, I find the contribution somewhat limited.

  • A related concern is the apparent lack of engagement with the extensive literature on preconditioning matrices in first-order optimization methods. In particular, semidefinite matrices can be multiplied with weight matrices to improve the efficiency of a network. This existing direction seems relevant to the approach proposed in the paper, but the authors seem to have come to these conclusions from a different line of research.

Questions for authors

I would be interested in the authors' perspective on the following intuition.

When an iterative optimization process addresses a single coherent task, weight decay may naturally emerge as part of the optimization dynamics. However, when a task involves several relatively independent subtasks, different parts of the network may respond differently to weight decay. In particular, accommodating a new task might require larger weights rather than further weight decay.

Do the authors consider this intuition consistent with their observations, or is there a different explanation for the behavior they report?

3 comments

We thank the reviewer for the assessment and for the question. At times we feel we did not fully understand the reviewer question and comments, therefore we applied our best reading in the answers below. We would appreciate if reviewer could provide references to make the implied context specific.

W1. Novelty. We agree that learned scales themselves are not new, and, in fact, devote a long Appendix section 1 to position our work with respect to existing literature. While various special cases of LRMs were introduced before, we jointly explain their benefit under a single mechanism articulated by observations (O1)-(O3). Then we ask a specific - why do matrix updates fail to correct a harmful scale? We find an answer in terms of optimizer pathology mechanism, which is quite different from other explanation proposed in the literature: geometric norm growth from transverse updates (van Laarhoven, 2017; Hägele et al., 2026) or noise-driven norm growth (Kosson et al., 2024; Zuo et al., 2025). We believe these are substantial conceptual novelties. Our thorough and broad empirical studies support these conceptual contributions.

The framing of LRMs as "learning the learning rate" is indeed points to the role of LRMs as a optimizer preconditioner that directly scales the learning rate, and this framing is considered as the mechanisms behind empirical benefits of LRMs in Wang et al (2026). However, our experiments (fig.1 is the most direct evidence) point to the central role of LRM as in observation (O3): correcting the harmful scale of the underlying matrix stuck in suboptimal noise-WD equilibrium.

W2. Preconditioning literature. We would again appreciate specific references. We see two possible readings.

Preconditioned optimizers such as K-FAC (Martens & Grosse, 2015) and Shampoo (Gupta et al., 2018) multiply the gradient by positive semidefinite matrices. This changes the optimizer and is complementary to our work, which changes the parameterization. LRMs can be combined with any optimizer; we evaluate them with Adam and with Muon, which is closely related to Shampoo, and find gains with both (Fig. 8).

Reparameterization as preconditioning. Under gradient flow, learning a scalar s jointly with W gives, for W=sW and w=vec(W),

ddtvec(W‾)=−(s2I+ww⊤) ∇W‾L,\frac{d}{dt}\mathrm{vec}(\overline{W})=-\big(s^2I+ww^\top\big)\,\nabla_{\overline{W}}L,

a positive semidefinite preconditioner. We discuss this view in App. A.3. It explains faster convergence in settings where direct matrix training also converges. It does not explain O1, where the matrix update moves the scale in the wrong direction. Under Adam the equivalence also breaks: the scalar receives the aggregated gradient ⟨W,G⟩\langle W,G\rangle before normalization, so it changes the statistics the optimizer sees, not only the geometry (App. A.4, I.8).

Question. Subtasks and weight decay. We have two reading of this question.

The first reading. When a model serves several fairly independent subtasks, the parts of the network that serve them may need different weight scales, and a uniform weight decay may hold some of them smaller than they should be. Under this reading, we find the intuition largely consistent with our observations: weight decay holds the rows of a matrix near a common scale, ∥W∥∝η/λ\|W\|\propto\sqrt{\eta/\lambda}, whereas different features prefer different scales. Without LRMs the row norms are tightly concentrated; with vector LRMs they spread over a much wider range (Fig. 5), and learned scales also vary strongly with depth (Fig. 4). In the reviewer's picture, if a subtask relies on particular features, a row or column multiplier gives each of those features its own scale. That scale is learned from the task gradient, so it is set by what the training objective requires of the feature and not by ηλ\frac{\eta}{\lambda}.

The second reading. Adding more subtask might require larger norms, as if each new subtask ana_n requires adding its own term to the weights Wn=Wn−1+WanW_n = W_{n-1} + W_{a_n}. Then weight decay would prevent the growth of the norm as expected from WnW_n, limiting the performance. Our understanding actually does not agree with the above interpretation: proportionally growing the norms of all the matrices in the model actually has a weak impact on the model performance, in part by presence of normalization layers and related scaling symmetries we discuss in App. E.1. Therefore, even the version of WnW_n downscaled by weight decay should perform relatively close. Our intuition is closer to another version of the accommodation argument, where independent subtask have orthogonal direction in the weight space of fixed dimension, and hence the capacity of the model to fit many independent task is limited by the dimension of of its weights. This would explain, for example, why small language model can achieve good performance on narrow tasks, while generalizing over many of them require large models.

Thank you for the detailed response.

To clarify my original comment on novelty, I was not suggesting that learning multipliers is mathematically equivalent to learning the learning rate. My point was broader: adapting quantities that influence optimization dynamics, including learning rates, scaling factors, and preconditioning matrices, has a substantial history in the literature.

I appreciate your explanation of how the proposed multipliers differ from conventional preconditioning, particularly under Adam, and your emphasis on correcting potentially harmful scale equilibria arising from the interaction between optimizer dynamics and weight decay.

Your response to my question about multiple subtasks is also helpful. The observation that different features may require different scales is consistent with the intuition I raised.

My remaining concern is the extent of the conceptual novelty beyond these established ideas. While the proposed explanation involving noise–weight decay equilibrium is interesting, I am not yet convinced that its broader explanatory power has been established sufficiently to distinguish it as a substantial new contribution. The empirical investigation is extensive and useful, but the theoretical mechanism appears more narrowly supported than the range of experimental settings considered.

Overall, the rebuttal has clarified the paper's contributions and their relationship to existing approaches. However, it has not sufficiently resolved my main concern regarding novelty and significance. I will therefore maintain my current score of 2.

We thank reviewer, for the detailed comment!

The narrowed down novelty concern has a substantial interaction with the reviewer BZ1S concern W1, and we respectfully encourage the reviewer to look at our fresh response to that concern. We would like to take an opportunity to reiterate and further develop our novelty proposition on top of that response.

At high level, the main thread of our paper is the following:

  1. LRM demonstrate practical benefit to model pretraining, as supported both by our experiments results, results in earlier works on specific variants of LRM (e.g. ReZEro and LayerScale), or in the close concurrent works with essentially full LRM configurations (Hägele et al., 2026, Wang et al., 2026).
  2. We connect the benefit of LRMs to inability to correct harmful scales, as articulated by our observations O1-O3.
  3. We find a concrete mechanism of failure to correct harmful scale (O1) in LM head - rare event suppresion/reversal of Adam drift - and demonstrate it both theoretically and experimentally.
  4. Our experiments (specifically MLP experiment in App. G) shows that the same phenomenology applies to backbone matrices, though the mechanism behind stays open.

We believe the steps 2-4 of the thread above present a serious conceptual contribution, essentially a blueprint of a complete path from empirical intervention to its mechanistic understanding that connects improvement in practical training to the underlying training problem. This is why we emphasize the notion of optimizer pathology and dedicate a sec. 2.4 to it.

The following strategy seems viable to us: locate a certain optimizer pathology and then introduce a targeted weight reparametrization to correct it. For now, the only worked out example of such pathology is related to norm growth. But given that pathology mechanism is still unclear for backbone matrices, there is a possibility that whenever this mechanism is found, its impact might reach beyond abnormal norms growth - and hence not covered by current formulation of LRM which are design to only correct the harmful norms.

Hence, our observation O1-O3 + optimizer pathology is essentially a framework for locating and translating existing optimizer failure modes into model improvements. And we view this framework as the conceptual novelty of the current paper.

Reviewer_BZ1SOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why separately learned scales improve language-model training despite adding no representational capacity. It proposes learnable multipliers analyzes adam’s scale-learning failures in a zipf toy model, and demonstrates training-loss and downstream improvements across architectures and optimizers.

Critical strengths
  • The paper offers a useful explanation of how weight decay and separately learned scales interact. Controlled experiments and the zipf analysis help distinguish matrix-norm regulation from task-dependent scale learning.
  • The placement ablation uses 20 paired seeds and includes rmsnorm and residual-scale controls. Long-pretraining experiments further demonstrate benefits across dense, hybrid, and moe architectures, including both adam and muon.
Critical weaknesses
  • The theoretical mechanism is established in a simplified Adam setting, while backbone gradients lack the same rare-sign asymmetry. Its connection to the broader backbone and muon improvements therefore remains unresolved.
  • Each long-pretraining comparison uses only one baseline–lrm pair, and downstream gains are mixed on some tasks. Averaging correlated checkpoints does not establish robustness across independent training runs.
Questions for authors
  • Have you directly measured the mean adam update on frozen LM-head gradient streams to verify suppressed or reversed drift? The sign statistics alone do not establish that the theoretical conditions hold.
  • How sensitive is the proposed mechanism to unequal momenta and nonzero epsilon? The analysis assumes equal momenta and zero epsilon, whereas the practical experiments use different settings.
  • Can you provide independent-seed replication for a representative long-pretraining pair? This would help assess the reliability of downstream gains and the comparison with switching from adam to muon.
  • How do lrm compare empirically with normalized magnitude–direction methods discussed in Appendix A? A matched comparison would clarify the practical benefit of the unconstrained multiplier parameterization.
2 comments

We thank the reviewer for the careful reading, positive evaluation, and the concrete questions.

W1. Unexplained mechanism. We fully agree with the reviewer, and state it in the paper, that the mechanism of norm growth in backbone matrices, for both Adam and Muon, is not explained at the moment. However, we firmly believe that this is not a weakness but a clear boundary of the current paper scope.

Our mechanism is the answer to the observation O1 - the inability to control harmful matrix scale during native optimization. The common explanations in the literature of why the norm grows include:

  1. Geometric: for scale-invariant weights the gradient is orthogonal to W, so each step increases ∥W∥\|W\| without changing the function (van Laarhoven, 2017; Arora et al., 2019; Li & Arora, 2020; Wan et al., 2021; Kosson et al., 2024).
  2. Loss-driven: for homogeneous models with cross-entropy, growing the norm keeps reducing the training loss (Soudry et al., 2018; Merrill et al., 2021).

but neither accounts for our observation O1. From this point of view, our mechanism based on rare event Adam failure mode provides one clear example, supported by real LLM behavior and detailed understanding in toy setting. This makes our mechanism a conceptual contribution showing how the harmful matrix growth can be realized. We believe the generalization of this Adam mechanism - the optimizer pathology discussed in sec. 2.4 - is a productive and unexplored direction for understanding matrix dynamics during training. We tried to spend some time to find the specific realization of the pathology in backbone matrices, but it proved not so simple, therefore we left it out of the scope of this work as description such mechanism would presumably deserve its own paper.

Q1. Adam drift reversal on LM head. We have measured the mean Adam update on frozen LM-head gradient streams of the same checkpoint that was used for gradient sign statistics, and compared gradient direction vs Adam drift direction entry-wise, and observe strong evidence for the Adam drift reversal. Below are the fraction of LM head parameter entries with flipped Adam drift across the sign groups and with extra filtering to exclude statistically not resolved signs of drift and gradient; the fraction of entries in each cell is reported in (brackets).

All entriesWeakest 50% removed on both axes
Group A (rare sign carries the mean)76.1% (55.7%)84.3% (11.2%)
Group B (common sign carries the mean)2.2% (44.3%)0.007% (16.6%)

Specifically, we first break all the parameters in two groups: B is a normal parameter entries where the sign of the mean gradient coincide with the most frequent sign of individual gradients, A is the complementary rare event group with less frequent sign group of individual gradients determining the sign of the mean. The second column address one potential failure of sign based analysis - the entries whose mean gradient or drift are near zero and the respective sign is not informative. To exclude such entries which are either (i) have the relative mean gradient strength (E[g])2/E[g2](\mathbb E [g])^2 / \mathbb E [g^2] below the median over all LM head entries, or (ii) the absolute value of the Adam drift is below the median over all LM head entries (no normalization is needed since adam update is already dimensionless).

As expected from our theory, the drift reversal are frequent and dominate the rare sign group A and almost absent from normal sign group B; the rare sign group A itself contains more than half of LM head entries. Importantly, this effect can not be attributed to some aberration due to signs of statistically unresolved mean gradient or drifts, as our aggressive below median filtering shows (second column).

Q2. Sensitivity to unequal momenta and nonzero epsilon. Our derivations for the Zipf model and general Adam result in theorem 2.1 uses direct computation of various moments, and rare even structure of the gradient streams, that are insensitive to β1≠β2\beta_1\ne\beta_2 or ε>0\varepsilon>0.

We do expect qualitative change of results though in less interesting regions of parameter space:

  • If ∣log⁡1−β11−β2∣≫1|\log \frac{1-\beta_1}{1-\beta_2}| \gg 1 the nominator and denominator of Adam updates will have radically different width of memory windows, leading to distinct behavior. But such values give training instabilities in LLM training.
  • When ε\varepsilon is large enough, Adam transitions to SGD with momentum. The same transition removes convergence acceleration from Adam preconditioning, make it less interesting algorithm to study.

Within the reasonable range of β1,β2,ε\beta_1,\beta_2,\varepsilon, we expect results to not change qualitatively, while making the exposition much more bulky, defeating its purpose as a clear demonstration of the optimizer pathology phenomena (see W1 response).

W2 & Q3. independent-seed replication for long pretraining runs. We agree that averaging checkpoints of one trajectory does not measure run-to-run variability, and the independent seed is the most clear way to average the evaluation noise. However, we intentionally decided not to do such independent seed long pretaining replications (see also our comment in App. B.3.). We share our reasoning behind these choices below.

First, we note that our overall training budget is already high compared to the close works: 44B tokens in Hägele et al. (2026), whose models reach 1.29B parameters dense and 6.7B total (0.81B active) MoE, and roughly 100B tokens in Wang et al. (2026), whose models reach 1B dense and 2B total (0.62B active) MoE. Our pretraining runs are at larger scale and reach nontrivial eval scores on harder benchmarks. As a result, we were faced with a question of allocating the fixed budget across the experiments. independent-seed replication of long pretraining runs, which cumulatively take more than half of all the budget, would mean to compromise on one of the 3 axes: (i) reduction of training duration, (ii) model size, (iii) or removing pairs of tested architectures/optimizers. The first two axes would result in much weaker evaluation signal on the chosen benchmarks, and also validation of the method on smaller scale. The last axes would remove the diversity of tested settings which is of high value.

Then, we were aware of the benefit of independent seed replications and applied it selectively for the 20-seed placement ablation experiment. Here, on one hand, thorough averaging over independent seeds was required to separate signal from noise in measuring fine-grained impact of individual LRMs. On the other hand, independent seed replication was affordable due to shorter duration and smaller model size than in long pretraining.

Lastly, the temporal averaging of eval score along the special post LR decay continuation is the best way we found to meaningfully reduce the evaluation noise without blowing the training budget. And overall significant gap between LRM and non-LRM runs in each pair make us believe that the current settings are sufficient to claim overall improvement from LRMs.

Therefore, we remain confident in our allocation of training budget among our experiments.

Q4. Comparison with normalized magnitude–direction methods. The multipliers are unconstrained in both families. The difference is how the bare matrix norm is controlled: by a hard constraint inside the optimizer (e.g. Hägele et al., 2026) or by weight decay, which is the setting we study. Since AdamW and Muon with weight decay are the standard recipes, our question is why gains help there and what limits matrix-scale learning; LRMs are then a drop-in addition that leaves the optimizer unchanged.

Our results suggest the two families should behave similarly as far as scale adaptation is concerned: once gains are present, the loss is nearly insensitive to where the matrix norm sits (Fig. 1, right; App. G). Remaining differences would then come from how each method controls the angular update size, which is a separate effect. We agree a matched comparison would be valuable and will add this discussion to App. A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.