acceptodds
Under review as a conference paper at ICLR 2027

Estimating Optimal Hyperparameter Scaling Laws Efficiently through Power-Law Entropy Search

Abstract

Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expensive large-scale tuning. However, estimating these scaling laws conventionally requires exhaustive grid searches over thousands of training runs, consuming enormous computational resources. We introduce Power-Law Entropy Search (PLES), a computational cost-aware acquisition function built on multi-fidelity Bayesian optimization that efficiently estimates optimal hyperparameter scaling laws through adaptive experimentation. A key innovation in PLES is that it searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function. At each iteration, PLES selects the candidate configuration that maximally reduces the uncertainty of the scaling law estimates per unit computational cost, naturally favoring informative small-scale experiments. We evaluate PLES on synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs. Across all settings, PLES converges to accurate optimal hyperparameter scaling laws using less than one-tenth of the computational budget required by conventional grid search and other baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Optimization, Optimal Hyperparameter Scaling Law, Entropy Search

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_undbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper discuss how to efficiently fit the hyperparameter scaling law.

Critical strengths
  1. The paper focus on an important problem: estimating how to fit hyperparameter scaling laws with lower computational cost instead of propose another formula to fit the scaling law. This technique is very important as the model size becomes larger and larger with today's LLM and how to fit the law efficiently is needed.
  2. The method is clean, combining cross-scale GP modeling with a cost-aware acquisition function that directly targets scaling-law uncertainty. It is a really interesting direction to evaluate whether the law is fitted and what is the leaving information.
  3. The evaluation is very comprehensive, covering synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pretraining setup. The authors not only compare the fitting error but also compare the real performance of the solved hyperparamters.
Critical weaknesses
  1. The LLM training experiments fix D=20ND=20N, so it only validate a one-dimensional scaling relationship and do not verify the full NN and DD dependent scaling law. I understand it is costly to train more combination (D,N)(D,N) but I think it is needed.
  2. The method assumes a power-law functional form in advance. I find that the paper discuss this assumption in the limitation section but do not provides limited evidence on robustness when this assumption is misspecified. I think some experiments are needed to verify when this assumption do not satisfy whether the method will work.
Questions for authors
  1. Can the authors provide some LLM training experiments where NN and DD do not satisfy D=20ND=20N to validate the full two-dimensional scaling law on real LLM training?
  2. How robust is this method when the true optimal hyperparameter scaling law deviates from a power law, and can the posterior uncertainty become small despite systematic extrapolation bias? Maybe some ablations are needed. You can design some non-power-law synthetic experiments.
2 comments
AuthorsOpenReview ↗

Response to Reviewer undb

Thank you for taking time to review our paper. We address your questions and comments below, and would appreciate your consideration in raising your score in light of these clarifications and new results.

How robust is this method when the true optimal hyperparameter scaling law deviates from a power law. [...] You can design some non-power-law synthetic experiments.

We created a synthetic benchmark where the optimal hyperparameter follows a power-law with a curvature applied: \begin{align} \eta^\star(N,D) &= 0.1896,N^{-0.734}D^{0.342} 10^{\frac{\delta}{2}\left(u_N^2+u_D^2\right)}, \ u_N &= \frac{\log_{10}N-8.3}{0.7}, \ u_D &= \frac{\log_{10}D-9.3}{1.7}, \qquad \delta=0.15. \end{align}

This expression looks complicated but basically, the δ\delta term simply controls how much the function deviates from a power-law curve. After running all baselines, we see that PLES is still more efficient than other baselines, showcasing its effectiveness in finding optimal hyperparameters, even when the scaling-curve function is misspecified.

Original benchmark / search settingPLESSobolLadder BOGrid search
Synthetic0.09× (9%)Not convergedNot converged1.00×
Surrogate0.12× (12%)1.00×0.77×0.65×
LLM — 200M0.21× (21%)1.00×0.86×0.56×
LLM — 600M0.56× (56%)1.00×0.89×0.68×

Just for additional information, we would also like to point the reviewer to Fig. 9(b) that we have added to the revised PDF. It shows that as we increase the δ\delta term, the scaling law deviates further from power-law and the convergence of PLES gets gentler as the underlying scaling law is no longer power-law (which is expected). However, we believe some further research in this area can help compensate for this limitation. For instance, one promising research area is to come up with a scaling law family that can cover a wider range of function choices (with coefficients that control which scaling law function it becomes).

Can the authors provide some LLM training experiments where N and D do not satisfy D = 20N to validate the full two-dimensional scaling law on real LLM training?

Thank you for the suggestion. Due to time and budget constraints, we are unable to fully finish pre-training the LLMs from scratch during the rebuttal phase (we hope you find this reasonable). However, we would like to point out that our surrogate experiments are built from real LLM training runs that vary both model size and data size [1].

In addition, for the LLM pre-training runs in our experiments, we follow standard practice in the literature where we typically vary scaling ladders that follows a fixed tokens-per-parameter (TPP) ratio, usually with compute-optimal D ≈ 20N from Chinchilla [2]. A few examples of existing technical reports are as follows: Cerebras-GPT [3] trains models from 111M to 13B parameters at 20 TPP, and Gadre et al. [4] build scaling ladders at fixed token multipliers M = D/N. Recent work on hyperparameter scaling also expresses the optimum in terms of TPP [5]. Last but not least, our theoretical formula still works exactly the same even if we adjust the training data tokens to be proportional to model scale (i.e., Eq.3 still holds true even if we vary only N). As such, we feel our current existing results are comprehensive enough to showcase the effectiveness of PLES. These are fruitful discussion points, and we will add them to our revised manuscript.


Once again, thank you for taking time to review our paper. We hope we had addressed your questions, and improved your opinion of our work. We have uploaded a revised version of the PDF (extra information on page 10, in red).

[1] Li et al. Predictable Scale: Part I, Step Law – Optimal Hyperparameter Scaling Law in Large Language Model Pre-training. 2025.

[2] Hoffmann et al. Training Compute-Optimal Large Language Models. 2022.

[3] Dey et al. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. 2023.

[4] Gadre et al. Language Models Scale Reliably with Over-Training and on Downstream Tasks. 2024.

[5] Bergsma et al. Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pretraining. 2025.

Thank you for the clarifications and additional results. Your response has addressed my concerns, and I will maintain my positive score.

Reviewer_mikTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Power Law Entropy Search (PLES), a method for estimating optimal hyperparameter scaling laws with fewer training evaluations. The method uses a Gaussian process to model validation loss across model sizes, data sizes, and hyperparameter values. It then estimates optimal hyperparameters through Thompson sampling and fits power laws using Bayesian linear regression. New evaluations are selected to reduce uncertainty in the scaling law while accounting for computational cost. The method is tested on synthetic functions, a GP surrogate based on prior training runs, and real Llama style pretraining experiments. The authors report that PLES estimates accurate scaling laws using about 10% of the budget required by standard search methods and can also recover scaling behavior for hyperparameters that are not directly targeted.

Critical strengths
  1. The problem matters. Fitting hyperparameter scaling laws for large language models can cost close to a million GPU hours. A method that selects experiments specifically to improve the estimate of the law could help many groups.

  2. The core idea is natural and well suited to the problem. PLES targets the uncertainty of the scaling law itself rather than the optimum at a single scale. One GP shares information across scales, which avoids fitting every scale independently. The cost term also pushes queries toward cheap scales, and Figure 8 shows that most queries indeed cost less than 2% of a run at the target scale.

Critical weaknesses
  1. The diagonal uncertainty approximation in Eq.(6) needs further justification. The estimated optima at different scales come from a shared GP posterior, while Λ\Lambda uses only their marginal variances. The paper does not show whether this approximation adequately captures uncertainty in the scaling law coefficients.

  2. Several implementation details are unclear in the main text. The GP kernel, input transforms, and hyperparameter fitting procedure are not specified. Although the GP equations include a noise parameter, its value or estimation procedure is not reported. The experimental values of dd, KK, and CC, the number of initial points, and the stopping thresholds are also unclear. These omissions make the experiments difficult to reproduce, particularly because the kernel controls how information is shared across scales.

  3. The real LLM experiments fix D=20ND=20N and vary only model size. This setting tests a single scaling path and cannot separate the effects of model size and data size. It therefore provides limited evidence for the proposed scaling law when the two scales vary independently.

Questions for authors
  1. The baseline comparison does not fully isolate the contribution of the PLES acquisition. Grid search and Ladder BO do not share information across scales. Sobol uses the same shared GP and scaling law estimation procedure, but it does not select experiments adaptively. Could you compare PLES with another adaptive acquisition using the same GP and estimator under the same computational budget? It would also help to separate the effects of the entropy objective and the cost term.

  2. Could you provide a table comparing the number of training runs, cumulative computational cost, and achieved loss of PLES and direct HPO at each target scale? Please report the cost ratio at the same loss threshold so that readers can verify the claim that PLES uses less than 10% of the computational budget.

  3. Figure 1 and Figure 6 look like the same image. Could you provide the experimental settings for both figures and clarify whether they come from the same experiment? If so, why is the same figure shown twice?

  4. Why do you use the log determinant of Σw\Sigma_w rather than the variance of the prediction at the held out scale? Could you compare the two criteria?

  5. What exactly converges in the claimed asymptotic result? Is it the approximation of a fixed posterior quantity or the recovery of the true scaling law after more experiments? Does it still hold with the cost term?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mikT [1/2]

Thank you for taking time to review our paper. We address your questions and comments below, and would appreciate your consideration in raising your score in light of these clarifications and new results.


The diagonal uncertainty approximation in Eq.(6) needs further justification.

This diagonal uncertainty represents the observation noise associated with each observation in the Bayesian linear regression model. In the literature [1,2], the observation noise is typically treated as independent (it is usually too data-demanding to infer correlated observation noise). In our data-scarce setting, we believe it is rational to model these observation noise independently. Note, however, that this modeling assumption is about the independence of observation noise, not about the observations. Our GP model can and will still capture the dependencies between observations and produce nontrivial posterior covariances.


The GP kernel, input transforms, and hyperparameter fitting procedure are not specified.

Thank you for flagging this. Here are the implementation details, which we have missed in the original manuscript: We use a SE-kernel throughout our work (because of the relatively smooth hyperparameter-loss landscape of LLMs - see Fig. 2 and [1]), we use the standard normalization input transformation, and update the hyperparameters of the GP every iteration with the API in BoTorch via maximum-likelihood estimation. We used d=1d=1, K=8K=8, and C=128C=128 in our paper (note that we did not cherry-pick these values, and we believe our method’s overall effectiveness is fairly independent of these choices. d=1d=1 is chosen because it reduces our acquisition function to the reduction of uncertainty per unit compute-cost. Lastly, we stop our experiments when we do not observe large changes in the estimated scaling curve. A more detailed discussion on the stopping criterion can be found in Sec. 5.4 titled “Stopping Criterion”. We have added these implementation details to the revised version of the paper (see page 10, in red).


The real LLM experiments fix and vary only model size.

Thank you for the comment. We agree that validating the full two-dimensional scaling law is important. However, our evaluation already does so on real training data. The surrogate benchmark is constructed from real LLM pre-training runs [1] that vary model size and data size independently, so PLES is evaluated against real (N, D) loss surfaces rather than only synthetic ones. The fixed-TPP ladder used in our direct pre-training experiments is chosen because it reflects how scaling ladders are typically built in research and practice. In fact, when training production-level models (i.e., see the technical reports [2-5]), we often vary model scale but fix the amount of training token to be proportional to the model scale. As such, we believe our experiments are consistent with what is done in practice. We will add these discussion points into our revised manuscript.


Could you compare PLES with another adaptive acquisition using the same GP and estimator under the same computational budget?

Thank you for the question. We would like to emphasize that PLES is the first adaptive algorithm that explicitly learns the optimal hyperparameter scaling curve. We do not believe there exists a clear alternative acquisition function that competes with our method. This is exactly what makes our approach novel in the first place.

For completeness, we performed additional experiments to compare our method with SL2 [6], the closest adaptive method that we are aware of (however, it is actually not suitable for our setting, as described in our paper). The results are shown in Fig.9(c) and 9(d) of the revised manuscript and shows that PLES still enjoys better convergence. If you have any suggestion for what other adaptive acquisition function to use here, please let us know!

AuthorsOpenReview ↗

Response to Reviewer mikT [2/2]

Could you provide a table comparing the number of training runs, cumulative computational cost, and achieved loss of PLES and direct HPO at each target scale? Please report the cost ratio at the same loss threshold so that readers can verify the claim that PLES uses less than 10% of the computational budget.

Here is the table showing the results from Fig. 5. Note that Fig. 5 uses a log-scale for the x-axis so the difference might appear small in the original paper (we will make that more clear in the figure). Here, we used the worst baseline as the reference point and show the fraction of compute necessary to reach the same converged performance of every baseline.

Original benchmark / search settingPLES (ours)SobolLadder BOGrid search
Synthetic0.09× (9%)Not convergedNot converged1.00×
Surrogate0.12× (12%)1.00×0.77×0.65×
LLM — 200M0.21× (21%)1.00×0.86×0.56×
LLM — 600M0.56× (56%)1.00×0.89×0.68×

Note that in the synthetic and surrogate case, PLES only needs ~10% of the budget necessary to reach the converged model loss.

We agree that our abstract might have made a slight overclaim due to a missing phrase. We will revise it to "…to reach the same optimal loss at the held-out scale, PLES requires as little as 10% of the compute budget used by competing baselines…". Our method uses as little as ~10% of the compute budget in the best-case scenarios we considered (this holds true in the synthetic and surrogate cases). In other settings, it uses around 20-40% of the compute budget required by other baselines (which implies we are still more efficient than other baselines). Thank you for pointing this out, and we will carefully revise our paper to be more precise about our claims.

Figure 1 and Figure 6 look like the same image.

They are the same image. We purposely, for clarity and exposition purposes, used Figure 1 at the beginning to give readers a clear picture of what our algorithm is doing (we also feel the change in the scaling law curvature and uncertainty gives a good idea of what we are attempting to do in the paper).


Why do you use the log determinant of rather than the variance of the prediction at the held out scale?

Log-determinant is able to capture the overall uncertainty associated with our scaling-law coefficient estimate. This allows us to explicitly perform experimentation to reduce uncertainty of the overall scaling-law. (a side note: we believe this is highly correlated with the variance of the prediction at the held-out scale anyways). One major drawback with using the variance of the prediction at the held-out scale is that during experimentation, we might not know what is the held-out scale (i.e., the size of the model we want to actually train and deploy). We will add this information in the revised manuscript. Thank you for pointing this out.


What exactly converges in the claimed asymptotic result?

There are two metrics to measure convergence. First, we did a round of very fine-grained grid-search so we know the ground-truth optimal hyperparameter for all synthetic, surrogate and LLM-pretraining experiments. As such, we know how close we converge to the optimal hyperparameters in our experiments. Second, we can also measure the optimal model loss that our method produces at the largest held-out scale (based on the optimal hyperparameter we estimate). If this model loss becomes close to the ground-truth optimal model loss, it also implies that we have converged to the optimal hyperparameters.

Of course, in practice, we wouldn’t know the ground-truth optimal hyperparameters (that is what we want to find out). Rather, our algorithm “converges” when our uncertainty of the scaling law estimate becomes small. This is exactly the “stopping criterion” that we mentioned in Sec. 5.4. The visualization of our scaling-law estimate becoming more certain can also be seen in Fig. 6 and 7.


Once again, thank you for taking time to review our paper. We hope we had addressed your questions, and improved your opinion of our work. We have uploaded a revised version of the PDF (extra information on page 10, in red).

[1] Li et al. Predictable Scale: Part I, Step Law – Optimal Hyperparameter Scaling Law in Large Language Model Pre-training. 2025.

[2] Hoffmann et al. Training Compute-Optimal Large Language Models. 2022.

[3] Dey et al. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. 2023.

[4] Gadre et al. Language Models Scale Reliably with Over-Training and on Downstream Tasks. 2024.

[5] Bergsma et al. Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pretraining. 2025.

[6] Li et al. Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

Thank you for the detailed response. The additional results and clarifications have addressed my concerns and improved the overall quality and credibility of the paper. Therefore, I will raise my score from 2 to 3.

Reviewer_yiKQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PLES, an algorithm to estimate training hyperparameters at a larger, held-out scale with low compute. The GP models loss is used as a function of model size, data size, and hyperparameters. Thompson samples from the GP give the optimal candidate and its uncertainty at each scale, and the estimates are fitted to a power law using Bayesian regression. A cost function then selects the next experiment that most reduces the uncertainty of the power law coefficients per unit cost. Experiments on a synthetic function, a surrogate to real training runs are rolled out

Critical strengths
  1. The problem is practical. Tuning hyper-parameters for large models is expensive, and sharing information across different scales makes sense.
  2. The method is well designed and easy to follow. It connects the GP, the per-scale optima, and the power-law fit in a principled way.
  3. The method takes experiment cost into account. Experiments run at cheap scales.
  4. The method provides a natural stopping criterion based on uncertainty at the target scale.
Critical weaknesses
  1. Grid search and Sobol are non-adaptive, and Ladder BO optimizes each scale independently. None of them targets the scaling law itself, which is the key idea of PLES. Li et al. (2026), discussed in Section 2 as the closest work, is not compared. Even a small comparison in the surrogate setting, plus a multi-fidelity BO baseline, would better support the claimed gains. If this is not feasible, please explain why.

  2. The 10% claim is ill-defined, and the real-LLM results don't support it.

The abstract says PLES is "10% of the computational budget required ... and other baselines." But Section 6.2 gives all four methods the same total budget, so this isn't something the experiments measure. The only place a 10% figure is derived is Section 6.6, where it compares PLES to performing HPO directly at the held-out scale ,not to any baseline. These are different comparisons, and the paper's own Contribution 3 states them separately and without a multiplier. Also, the "5 runs" denominator is asserted rather than justified; the 10% follows directly from it.

On the real LLM experiments, PLES's advantage is small. In the 600M held-out panel Ladder BO reaches the best attainable loss at least as early as PLES, and in the 1B panel both Ladder BO and Sobol do, with the error curves for all three overlapping over the second half of the budget. Clear separation appears only in Synthetic and Surrogate. This is the exactly the settings where the ground truth is a power law by construction

There is no stated criterion by which to check the convergence claim.

Questions for authors
  1. Which comparison does the abstract's 10% refer to? How is the 10% cost advantage over the baselines computed?
  2. By what criterion does PLES "converge much more quickly"?
  3. How robust is PLES when the power-law assumption does not hold? In the synthetic benchmark, the optimal learning rate and batch size follow exact power laws by design. The surrogate setting is similar case. These settings exactly match the assumptions. We can see the clear gains in Figure 5, while the advantage is smaller on real LLMs. Testing a setting where these assumptions do not hold would help clarify the method’s robustness.
2 comments
AuthorsOpenReview ↗

Response to reviewer yiKQ

Thank you for taking time to review our paper and the positive feedback. We address your questions and comments below, and would appreciate further discussions with respect to these clarifications and new results.


By what criterion does PLES "converge much more quickly"?

We measure the convergence by determining how much computational resources each baseline incurs before finding the optimal hyperparameter at the held-out, largest scale. This is visible in Fig.5, which shows that good method like PLES (ours) will reach a lower model loss or % LR error while incurring less cumulative computational resources (indicated by the x-axis). This is highlighted by the horizontal gap between our method and other baselines. Note that the x-axis is on a log scale (this isn’t immediately obvious, we will update the axis labeling to make that more clear).


Which comparison does the abstract's 10% refer to? How is the 10% cost advantage over the baselines computed? [...]

All of our baselines estimate the scaling law iteratively by performing experimentation, which incurs increasing resources. Fig.5 shows that for the synthetic and surrogate case (which comes from real experimental data from Li., 2025), our method uses 10% of the compute budget required by other baselines to reach the same optimal model loss at the held-out scale. This is shown by the horizontal gap between PLES and other baseline’s convergence plots.

We would like to note that there is a slight over claim in our abstract currently, and we want to rectify it. We will revise it to "…to reach the same optimal loss at the held-out scale, PLES requires as little as 10% of the compute budget used by competing baselines (i.e., up to a 10× reduction)…". This means that our method uses 10% of the compute budget in the best-case scenarios (this is true in the synthetic and surrogate cases). That said, we would like to emphasize that in the real-LLM training case, our method still substantially outperforms other methods (again, note that the x-axis shows a log-scale).


How robust is PLES when the power-law assumption does not hold?

Thank you for raising this point. We performed an additional round of experiment by creating a synthetic dataset where the optimal hyperparameter scaling law follows the following function (essentially it adds a curvature to the log-linear scaling function). Please see the results in page 10 of our revised PDF file (in red) - in Fig 9(a) and 9(b), we created a function that adds a curvature to the power-law (controlled by delta). We can see that PLES still have a higher convergence as compared to other baselines. In fact, we still take 3 times less resources as compared to grid-search.

MethodProportion of compute to reach held-out loss of 2.4 (within 10% of optimal model loss)
PLES (ours)0.31x
Sobol0.62x
Ladder BO0.92x
Grid search1x (worst method)

Li et al. (2026), discussed in Section 2 as the closest work, is not compared. Even a small comparison in the surrogate setting, plus a multi-fidelity BO baseline, would better support the claimed gains. If this is not feasible, please explain why.

Thank you for the suggestion.

We have run an additional experiment where we compared SL2 (as outlined in Li et al., 2026) to our method. The results are shown in Fig.9(c) and 9(d) of the revised manuscript and shows that PLES enjoys better convergence as compared to SL2 (around 2-3x faster).

Why do we perform better? As discussed in Sec. 2, SL2 (the method in Li et al. 2026) is built to fit loss scaling laws: given losses observed at smaller scales, it predicts how loss changes as scale grows. On the contrary, our method fits optimal-hyperparameter scaling laws: it predicts how the best hyperparameter configuration shifts with scale. Hence they address related but orthogonal directions. In the loss scaling, the goal is to figure out how model loss changes with increasing scale. In contrast, in our setting, we need to find the optimal hyperparameter at each scale by efficiently sharing information across multiple scales; we just have to ensure that we perform sufficient experimentation to model how the optimal loss-region (w.r.t. hyperparameters) changes across scales accurately, which incurs far less compute resources.

This mismatch in setting shows up in Li et al.'s own results as well - In Table 3 of their paper, they applied SL2 to predicting optimal hyperparameters, and the fit is very poor, with the R² close to 0, while their loss scaling laws reach R² values close to 1. As such, this shows that SL2 is not a suitable candidate for the optimal-hyperparameter setting in our paper.


Once again, thank you for taking time to review our paper. We hope we had addressed your comments and questions, and improved your opinion of our work. We have uploaded a revised version of the PDF (extra information on page 10, in red).

Thank you for the clarifications and additional results. I will maintain my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of discussion

As the discussion period comes to a close, we thank all reviewers for their effort in providing constructive, and positive feedback/scores. We had a fruitful discussion with the reviewers during this period. We would like to summarise these discussion points/additional results below:

  1. Reviewer yiKQ pointed out that our method is "well designed and easy to follow" and that "the problem is practical". We discussed the setting where the underlying scaling-law deviates from a power-law form. We showed, through empirical experiments, that our method is robust and remains effective in such settings as compared to existing baselines. We added these results to our revised manuscript in red (Fig. 9(a) and 9(b) in Sec. 8).

  2. Reviewer mikT remarked that our problem setting is relevant and our method is "natural and well suited to the problem". They advised us to compare PLES with SL2 introduced in (Li. et al., 2026). We added these additional results in Fig. 9(c) and 9(d) in our revised manuscript. We also elaborated and explained why SL2 is not well-suited to learn hyperparameter scaling laws (which is an orthogonal problem to fitting conventional loss scaling laws).

  3. Reviewer undb said that our "paper focus on an important problem", our "method is clean", and our "evaluation is very comprehensive". They asked us what happens if the underlying scaling-law does not follow a power-law form. We answered their questions with additional empirical results, which we added to Fig. 9(a) and 9(b) in Sec. 8. The results showcase that our method remains effective in such a setting. We believe that coming up with an algorithm that can simultaneously learn the scaling-law function family and fitted coefficients is a promising future research direction in this area.

We are glad to see that the reviewers are satisfied with our responses. We have included these fruitful discussions to Sec. 8 of our revised manuscript (in red). Once again, we thank the reviewers for their feedback which has improved our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.