acceptodds
Under review as a conference paper at ICLR 2027

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

Abstract

Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has been extensively studied, the scale vector remains poorly understood despite its ubiquitous use. In this work, we present a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure. First, we show empirically that although scale vectors constitute only a negligible fraction of model parameters, removing them substantially degrades LLM pre-training. Our theory further shows that, in Pre-Norm architectures, scale vectors do not increase expressivity; instead, they improve optimization through a self-amplifying preconditioning effect on subsequent linear mappings. Second, we investigate the role of weight decay for scale vectors. By distinguishing Input-Norm and Output-Norm layers, we theoretically show that weight decay is beneficial for the former but harmful for the latter, due to their distinct roles in optimization and expressivity. Third, motivated by this understanding, we propose three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-direction reparameterization. Both theory and experiments show that each improvement yields consistent gains. Finally, we combine these improvements into a unified scale-vector strategy and evaluate it through extensive LLM pre-training experiments on dense and mixture-of-experts models ranging from 0.12B to 2B parameters, across multiple optimizers and learning rate schedules, under industrial-scale token budgets. The unified strategy consistently achieves lower terminal loss than well-tuned baselines and exhibits more favorable scaling behavior, while adding negligible parameter and computational overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM pre-triaining, scale vector

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W5YoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why normalization scale vectors, despite their small parameter count, materially affect LLM training. Through theoretical analysis and controlled experiments, it identifies how scale placement, parameterization, and weight decay influence optimization, and turns these findings into practical training recommendations.

Critical strengths
  1. The distinction between input-side and output-side scales provides useful guidance on applying weight decay.
  2. Component ablations, parameter-count controls, and runtime measurements strengthen the empirical evidence.
Critical weaknesses
  1. The theory relies on simplified linear models and specific training dynamics. Its connection to the observed gains in Transformers trained with AdamW or Muon remains insufficiently established.

  2. Most mechanism-level analyses use Llama-0.12B or Gemma-style 0.5B models, while larger-scale experiments mainly evaluate the combined strategy. It therefore remains unclear whether the individual findings generalize beyond the smaller configurations.

Questions for authors
  1. The final configuration changes both scale parameterization and normalization structure. Do the relative contributions of these components remain consistent at larger scales?
  2. How sensitive is the input-side versus output-side weight-decay recommendation to decay strength and optimizer choice? Does the same distinction hold beyond the tested training recipes?
  3. Does the degradation caused by removing scale vectors persist in larger models, such as 7B, and across different normalization architectures? This would help establish the generality of the central observation.
Reviewer_uqAAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies scale vectors in LLM normalization layers and explain how they help training. It proposes separate scales for different branches, new ways to place and learn them, and weight decay rules based on their location. Experiments on FineWeb-Edu show lower validation loss for dense models up to 1B and MoE models up to 2B, with small added cost and downstream gains on MoE-2B.

Critical strengths

1.The paper studies scale vectors, showing that these small parameters can have a large effect on LLM training.

2.The paper explains how scale vectors help training and why weight decay can help in some places but hurt in others.

Critical weaknesses

1.Downstream results are reported only for MoE-2B, where the gains are modest. No downstream results are provided for dense models, so it is unclear whether their lower validation loss leads to better task performance.

2.The loss gains for dense models are relatively small. For Dense-1B, Figure 7 suggests a reduction of about 0.01, but exact final values and variation across runs are not reported. All models are trained on FineWeb-Edu, so it is also unclear whether the gains depend on this training dataset.

3.The added value of HG in the final method is unclear. Figure 4 compares HG with the standard baseline, but does not compare the full method with the same method without HG. Therefore, the results do not establish whether HG still provides a benefit when DNP, OR, and IWD are already used.

Questions for authors

1.Most of the ablation studies are conducted on the 0.12B model. Could the authors comment on whether this is sufficient to support the conclusions?

2.Should the authors add experiments on a Dense-2B or Dense-7B model? Even training on 50B or 100B tokens would help show how the model converges.

Reviewer_vQd2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates learnable scale vectors in LLM normalization layers, whose role has received limited attention despite their widespread use. Removing these vectors degrades pre-training, and the authors analyze their optimization benefits through self-amplifying preconditioning in simplified theoretical settings. They distinguish Input-Norm and Output-Norm layers to motivate different weight-decay treatments, then combine individual weight decay (IWD), branch-specific heterogeneity (HG), dual normalized placement (DNP), and magnitude-direction reparameterization (OR) into a unified strategy. Experiments on FineWeb-Edu show lower terminal validation loss across dense Llama and LlamaMoE models ranging from 0.12B to 2B parameters, with additional validation under Muon and the wsd learning-rate schedule.

Critical strengths
  • Novel focus on an overlooked component, with a practical weight-decay guideline. The paper explains the optimization role of scale vectors through self-amplifying preconditioning (Theorem 2.2) and addresses differing weight-decay practices across open-source LLM implementations (Lines 164–168). Theorem 2.3 analyzes the benefit of weight decay for Input-Norm scale vectors (Lines 195–201), while a separate expressivity argument and Gemma experiments support excluding Output-Norm scale vectors from weight decay (Lines 183–186, Figure 2). This distinction provides a practical guideline for normalization layers with different structural roles.
  • Systematic ablations and broad pre-training validation. The authors evaluate the proposed components incrementally on Llama-0.12B (Figure 4) and validate the unified strategy across dense (0.12B to 1B) and MoE (0.25B to 2B) architectures. The training budgets are approximately 100 tokens per parameter for dense models and per activated parameter for MoE (Lines 354–358). Additional Muon and wsd experiments (Figure 9) broaden the tested training configurations, although scale vectors still use AdamW in the Muon setup (Lines 944–946).
Critical weaknesses

W1. Incomplete weight-decay ablations and limited sandwich-norm evaluation. Gemma lacks the condition disabling wd on both Input-Norm and Output-Norm scale vectors (Figure 2). The both-no-wd result in Figure 4 uses HG+DNP+OR, so it does not fill these gaps. The unified strategy is also evaluated only on Llama and LlamaMoE (Lines 351–353), leaving its effectiveness on Gemma's existing sandwich-norm architecture unverified.

W2. Wall-clock benefit and broader generalization remain unverified. Runtime profiling averages only 10 post-warmup training steps on Dense-1B (Lines 928–931), showing approximately 4% time overhead (Table 5). The pre-training loss curves are plotted against tokens or FLOPs (Figures 6–9), leaving the time required to reach a target validation loss unverified. Downstream evaluation covers only MoE-2B and reports an average score increase from 48.39 to 49.17 without repeated-seed results or confidence intervals (Table 6, Lines 1026–1030). The five-seed check reports only Dense-0.12B baseline validation losses (Lines 1014–1019), so it does not establish the statistical reliability of the downstream gains. Post-training is not examined, and experiments stop at 1.03B dense and 2.00B MoE parameters (Tables 3 and 4), leaving behavior in larger, wider, and deeper models unverified.

Questions for authors

Q1. Complete Weight-Decay Ablations and Sandwich-Norm Evaluation. Could you add the both-no-wd condition for Gemma-0.5B in Figure 2? The both-no-wd condition in Figure 4 uses HG+DNP+OR and therefore does not isolate these settings. How does the unified strategy perform on Gemma's sandwich-norm architecture?

Q2. Equal-Resource Wall-Clock Time Comparison. Beyond the 10 post-warmup steps measured on Dense-1B (Table 5), could you report validation loss against wall-clock training time under identical GPU resources for dense and MoE models? Please also report the inference latency overhead of the additional normalization layers in DNP.

Q3. Scalability to Larger Model Sizes. Given the dd-dependent amplification along the scale-vector magnitude direction derived under gradient flow (Lines 336–344), where dd is the scale-vector dimension, does the unified strategy maintain stability and consistent gains in wider and deeper models, such as 7B parameters or beyond?

Q4. Downstream Statistical Significance and Post-Training Behavior. Could you provide downstream results over multiple seeds or with confidence intervals for both dense and MoE models? Separately, do you have evidence on whether the optimization benefits persist during instruction tuning, or should this be treated as a direction for future work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.