- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies why scaling laws estimated from small language models can be unreliable and argues that insufficient hyperparameter tuning is a major cause.
It characterizes how hyperparameter sensitivity changes with scale and develops a methodology for small scale experiments, which is evaluated on Llama style language models and a pre norm versus post norm case study.
Critical strengths
The paper performs an extensive hyperparameter study across a wide range of model and data scales, including held out larger models for evaluating extrapolation.
The paper goes beyond showing that tuning matters and analyzes how hyperparameter sensitivity changes with scale, then connects these observations to a practical methodology for small scale experiments.
Critical weaknesses
The main findings are based on one Llama style model family, one pretraining dataset, and one optimizer setup, so it is unclear how broadly the observed decrease in hyperparameter sensitivity transfers to other training settings.
The estimated effective number of hyperparameters decreases with scale, but the noisy quadratic fits use different portions of the score distribution at different scales. It is therefore unclear whether this trend is robust to the choice of fitting threshold.
Questions for authors
The noisy quadratic fits use roughly the best 7 to 8% of configurations at the smallest scales, but up to 50% at the largest scales. Can the authors report whether the estimated decrease in effective hyperparameter dimension remains when using the same tail fraction across scales, or provide a sensitivity analysis over a common range of thresholds?
3 comments
Thank you for the detailed rebuttal and additional experiments, which substantially address my original concerns. I will therefore raise my rating from 3 to 4.
Thank you for raising your score! Your feedback made our analysis more precise and broadened our experiments, and we included the new results in the revised manuscript. We appreciate your helpful and constructive review.
Response to Reviewer 1mSa
Thank you for the positive review and for your precise and actionable questions. Following your suggestions, we ran the threshold analysis you proposed with about 900 configurations per scale from 4M to 537M, and new experiments with a different dataset and a different model family. We respond to your question first, then address your general concerns.
Q1: Effective number of hyperparameters at a common threshold
Thank you for this insightful question. Following your suggestion, we repeated the analysis with common thresholds across scales, using the final training loss of about 900 configurations per scale from 4M to 537M.
Our findings in Section 4.1 give a good explanation for why one might set a higher quantile for the threshold in larger models: as models scale, a larger fraction of hyperparameter configurations fall near the optimum. Imagine instead that we defined the threshold in terms of how suboptimal a config is, then the quantile must naturally increase. For example, the 10th percentile at 4M parameters is 0.21 BPC worse than the optimum. At 537M parameters, the 10th percentile only extends .03 BPC above the optimum: 2.5 times the seed noise (σ≈0.012 BPC). This lower signal-to-noise ratio can make the distribution more challenging to estimate.This is why our original percentiles grew with scale.
Below, we fix the threshold in terms of the degree of suboptimality. We show γ for this common window above the best loss:
With a common loss threshold, the effective number of hyperparameters still drops from two to one as models grow, for both designs.
Independently of any threshold, sensitivity also decreases strongly with scale. Using the same runs, below are the share of configurations within 0.1 BPC of the best and the gap between the 25th-percentile and best score, all in bits per character (BPC):
We see that more runs fall near the optimum as the scale increases indeed.
One model family, dataset and optimizer
Our study is model-centric, so we deliberately kept the dataset fixed while changing the model. Following your suggestion, we ran two new sets of experiments with the same 128 hyperparameter configurations at every scale from 4M to 134M, using (1) the same model family on a different dataset, DCLM-baseline [1], and (2) a different model family on the same dataset, a recurrent architecture built from Hawk's RG-LRU blocks [2]. Below is the gap between the 25th-percentile and best final training loss (nats), and the share of configurations within 0.1 nats of the best:
In both new settings, sensitivity decreases with scale as in our main study. The gap shrinks about 3 times from 4M to 134M, and the share of near-optimal configurations grows 3 to 5 times.
Thank you again for your feedback, which made our analysis more precise and broadened our experiments. We included these additional results in the revised manuscript. We made a considerable effort to address all of your concerns, and believe the new results substantially strengthen our paper; we hope that you would consider raising your score.
References
[1] Li, Jeffrey, et al. "Datacomp-lm: In search of the next generation of training sets for language models." Advances in Neural Information Processing Systems 37 (2024): 14200-14282.
[2] De, Soham, et al. "Griffin: Mixing gated linear recurrences with local attention for efficient language models." arXiv preprint arXiv:2402.19427 (2024).