- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why normalization scale vectors, despite their small parameter count, materially affect LLM training. Through theoretical analysis and controlled experiments, it identifies how scale placement, parameterization, and weight decay influence optimization, and turns these findings into practical training recommendations.
Critical strengths
- The distinction between input-side and output-side scales provides useful guidance on applying weight decay.
- Component ablations, parameter-count controls, and runtime measurements strengthen the empirical evidence.
Critical weaknesses
-
The theory relies on simplified linear models and specific training dynamics. Its connection to the observed gains in Transformers trained with AdamW or Muon remains insufficiently established.
-
Most mechanism-level analyses use Llama-0.12B or Gemma-style 0.5B models, while larger-scale experiments mainly evaluate the combined strategy. It therefore remains unclear whether the individual findings generalize beyond the smaller configurations.
Questions for authors
- The final configuration changes both scale parameterization and normalization structure. Do the relative contributions of these components remain consistent at larger scales?
- How sensitive is the input-side versus output-side weight-decay recommendation to decay strength and optimizer choice? Does the same distinction hold beyond the tested training recipes?
- Does the degradation caused by removing scale vectors persist in larger models, such as 7B, and across different normalization architectures? This would help establish the generality of the central observation.