- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper discuss how to efficiently fit the hyperparameter scaling law.
Critical strengths
- The paper focus on an important problem: estimating how to fit hyperparameter scaling laws with lower computational cost instead of propose another formula to fit the scaling law. This technique is very important as the model size becomes larger and larger with today's LLM and how to fit the law efficiently is needed.
- The method is clean, combining cross-scale GP modeling with a cost-aware acquisition function that directly targets scaling-law uncertainty. It is a really interesting direction to evaluate whether the law is fitted and what is the leaving information.
- The evaluation is very comprehensive, covering synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pretraining setup. The authors not only compare the fitting error but also compare the real performance of the solved hyperparamters.
Critical weaknesses
- The LLM training experiments fix , so it only validate a one-dimensional scaling relationship and do not verify the full and dependent scaling law. I understand it is costly to train more combination but I think it is needed.
- The method assumes a power-law functional form in advance. I find that the paper discuss this assumption in the limitation section but do not provides limited evidence on robustness when this assumption is misspecified. I think some experiments are needed to verify when this assumption do not satisfy whether the method will work.
Questions for authors
- Can the authors provide some LLM training experiments where and do not satisfy to validate the full two-dimensional scaling law on real LLM training?
- How robust is this method when the true optimal hyperparameter scaling law deviates from a power law, and can the posterior uncertainty become small despite systematic extrapolation bias? Maybe some ablations are needed. You can design some non-power-law synthetic experiments.
2 comments
Thank you for the clarifications and additional results. Your response has addressed my concerns, and I will maintain my positive score.
Response to Reviewer undb
Thank you for taking time to review our paper. We address your questions and comments below, and would appreciate your consideration in raising your score in light of these clarifications and new results.
We created a synthetic benchmark where the optimal hyperparameter follows a power-law with a curvature applied: \begin{align} \eta^\star(N,D) &= 0.1896,N^{-0.734}D^{0.342} 10^{\frac{\delta}{2}\left(u_N^2+u_D^2\right)}, \ u_N &= \frac{\log_{10}N-8.3}{0.7}, \ u_D &= \frac{\log_{10}D-9.3}{1.7}, \qquad \delta=0.15. \end{align}
This expression looks complicated but basically, the δ term simply controls how much the function deviates from a power-law curve. After running all baselines, we see that PLES is still more efficient than other baselines, showcasing its effectiveness in finding optimal hyperparameters, even when the scaling-curve function is misspecified.
Just for additional information, we would also like to point the reviewer to Fig. 9(b) that we have added to the revised PDF. It shows that as we increase the δ term, the scaling law deviates further from power-law and the convergence of PLES gets gentler as the underlying scaling law is no longer power-law (which is expected). However, we believe some further research in this area can help compensate for this limitation. For instance, one promising research area is to come up with a scaling law family that can cover a wider range of function choices (with coefficients that control which scaling law function it becomes).
Thank you for the suggestion. Due to time and budget constraints, we are unable to fully finish pre-training the LLMs from scratch during the rebuttal phase (we hope you find this reasonable). However, we would like to point out that our surrogate experiments are built from real LLM training runs that vary both model size and data size [1].
In addition, for the LLM pre-training runs in our experiments, we follow standard practice in the literature where we typically vary scaling ladders that follows a fixed tokens-per-parameter (TPP) ratio, usually with compute-optimal D ≈ 20N from Chinchilla [2]. A few examples of existing technical reports are as follows: Cerebras-GPT [3] trains models from 111M to 13B parameters at 20 TPP, and Gadre et al. [4] build scaling ladders at fixed token multipliers M = D/N. Recent work on hyperparameter scaling also expresses the optimum in terms of TPP [5]. Last but not least, our theoretical formula still works exactly the same even if we adjust the training data tokens to be proportional to model scale (i.e., Eq.3 still holds true even if we vary only N). As such, we feel our current existing results are comprehensive enough to showcase the effectiveness of PLES. These are fruitful discussion points, and we will add them to our revised manuscript.
Once again, thank you for taking time to review our paper. We hope we had addressed your questions, and improved your opinion of our work. We have uploaded a revised version of the PDF (extra information on page 10, in red).
[1] Li et al. Predictable Scale: Part I, Step Law – Optimal Hyperparameter Scaling Law in Large Language Model Pre-training. 2025.
[2] Hoffmann et al. Training Compute-Optimal Large Language Models. 2022.
[3] Dey et al. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. 2023.
[4] Gadre et al. Language Models Scale Reliably with Over-Training and on Downstream Tasks. 2024.
[5] Bergsma et al. Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pretraining. 2025.