- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
NCPL introduces the concept of fine-tuning LLMs for the task of predicting the performance of pretraining beyond the dimensions typically covered by scaling laws. Using the example of a Qwen3-1.7B model applied to two datasets showing the performance of training runs, NCPL demonstrates good accuracy compared to Chinchilla.
Critical strengths
- NCPL generalizes scaling laws to new parametric dimensions and training configurations.
- Prediction performance is better or comparable to other methods.
- Well-written and easy-to-follow paper.
Critical weaknesses
- The paper establishes that NCPL performs better than baselines, but does not explain how it works or how it arrives at those results.
- Comparison against DL methods (TabM) and gradient boosting (CatBoost) needs further explanation.
- Unclear meaning of “ground-truth optimal” in some evaluation benchmarks. [See below]
Questions for authors
-
You state that “hyperparameter selection without specifying an explicit functional form a priori by modeling the pretraining loss from heterogeneous training logs”. I don’t think the paper motivates that abandoning such functional forms is a good direction. Wouldn’t they be useful for understanding training behavior - in addition to performing prediction? Couldn’t your approach be used to discover new functional forms?
-
The evaluation clearly established performance in loss prediction, but the paper does not explain how NCPL actually works. Is the model discovering new scaling laws? Are old laws being adjusted to the provided dataset? What does the reasoning look like for out-of-distribution predictions? I think these are very relevant questions.
-
Scaling laws like Chinchilla are already known and included in the model’s training data. Is it possible that NCPL simply learns a simple modification of that law for the StepLaw dataset (Figure 6)? Did you investigate what functional form is used during prediction? Is it possible that the model simply applies minor adjustments to known scaling data to better fit into the provided dataset?
-
While reading Section 4.1, my immediate question was about the coverage of different parameters between the two datasets: are parameters sampled regularly and consistently? Section 4.4 seems to provide a partial answer - on the StepLaw dataset, which seems to vary fewer parameters, NCPL is frequently outperformed by other methods based on gradient boosting and deep learning. Wouldn’t this imply that NCPL is most usable for sparse datasets with diverse configurations? Wouldn’t a better approach be to employ hybrid inference, mixing more classical methods and NCPL depending on the parameters, configuration, and settings?
-
According to the Figure 6 caption, the plot shows relative losses of NCPL and power law against the “ground-truth optimal”. However, in a few plots, you report a negative relative loss. Wouldn’t this imply that the predicted configurations are better than “ground-truth optimal”? Can you please explain that?
3 comments
Responses to the Reviewer osdd (continued)
Q5: the meaning of “ground-truth optimal”
We apologize for the confusion. For each setting, we define as the minimum observed loss among the configurations evaluated in the StepLaw LR-BS grid search. For a configuration selected by NCPL or the power-law baseline, its loss is estimated by evaluating a quadratic surface fitted to the observed sweep results at the predicted (LR, BS). We then calculate the relative loss as , where denotes the loss estimated by the quadratic fit.
Since the quadratic fit provides a smooth approximation to the discrete grid-search results, its predicted loss can occasionally be lower than the minimum observed loss , resulting in a negative relative loss. We will clarify this evaluation procedure in the revised version.
Overall, we hope our response has addressed your concerns. We would also be happy to address any further questions during the discussion period.
Dear Reviewer osdd,
As the rebuttal period is nearing its end, we would be grateful if you could let us know whether you have any remaining concerns that we could address before it closes.
Thank you again for your time and effort in reviewing our work.
Best, Authors
Responses to the Reviewer osdd
We thank the reviewer for recognizing NCPL’s ability to extend scaling laws to new dimensions and training configurations and its competitive predictive performance. Below, we address your concerns in detail.
We thank the reviewer for raising these questions. Understanding the mechanisms behind NCPL's predictions is an important direction. At its core, NCPL uses a Transformer-based neural network to learn configuration-to-performance mappings from a large collection of heterogeneous pretraining logs. Specifically, given the full training configuration, NCPL predicts a residual relative to the Chinchilla baseline. Thus, the Chinchilla prior provides the dominant scaling behavior with model and data sizes, while the Transformer learns configuration-dependent effects and their complex interactions.
As with many neural network-based approaches, its internal representations and prediction mechanisms are not directly interpretable. In particular, extracting an explicit symbolic scaling law or an interpretable computational circuit underlying its predictions is very challenging. What's more, a compact representation may not even exist for such a high-dimensional configuration space. We believe understanding these learned representations and mechanisms is an interesting direction for future work, which we leave for further investigation.
We do not think that NCPL's performance can be primarily attributed to memorizing or making minor modifications to Chinchilla's law. In particular, as discussed in Section 4.4 and Table 5, we also train NCPL from scratch, without relying on the pretrained knowledge of Qwen, and observe comparable performance on the StepLaw dataset. This suggests that the model can learn the configuration-to-performance mapping directly from pretraining logs.
We agree that explicit functional forms are valuable for both prediction and understanding training behavior. However, existing scaling laws typically focus on one or two hyperparameters, rather than jointly modeling a broader range of training configurations. For example, Step Law jointly optimizes LR and BS while keeping other optimizer settings, including WD, fixed, and obtains the following optimal LR scaling law [1]:
LR∗=1.79N−0.713D0.307.
In contrast, MAI-Base-2 jointly optimizes LR and WD, obtaining a different functional form [3]:
-1.51 + 0.175\log TPP -0.573\log(\mathrm{steps}).$$ These examples illustrate that the resulting scaling relationships can depend on which hyperparameters are jointly optimized, highlighting the challenge of specifying a single functional form a priori to capture complex interactions among heterogeneous training configurations. Our goal is therefore not to abandon explicit functional forms, but to **avoid imposing potentially restrictive assumptions a priori when modeling these complex interactions**. We also agree NCPL can serve as a useful tool for discovering new functional forms. By analyzing the learned response surfaces and interactions, NCPL can help identify potentially interesting scaling relationships. We will clarify this perspective in the revised version. > Q4: NCPL is most usable for sparse datasets with diverse configurations? Hybrid inference? We agree the point that the reviewer raised. The two datasets indeed have substantially different configuration coverage. The StepLaw dataset conducts fine-grained sweeps over learning rate and batch size. In contrast, Marin explores a broader range of optimizers and hyperparameters through coordinate-descent sweeps with a sparse coverage. Indeed, our results show that NCPL is particularly advantageous for high-dimensional and heterogeneous configuration spaces with relatively sparse coverage, as in the Marin dataset. Such settings are common in LLM development, where many training factors, including optimizers, learning rates, batch sizes, and weight decay, jointly influence pretraining performance, while exhaustively exploring their combinations is prohibitively expensive. We also agree that hybrid inference is a promising direction. We view our work as a first step toward modeling heterogeneous configuration spaces, and leave the systematic design of hybrid predictors to future work.