acceptodds
Under review as a conference paper at ICLR 2027

Configuration-to-Performance Scaling Law with Neural Ansatz

Abstract

Researchers build scaling laws to forecast the training performance of expensive large-scale runs with larger model size and data size . These laws assume that other training hyperparameters are optimally chosen, which can require significant effort and, in some cases, be impossible due to external hardware constraints. To improve predictability across a broader set of hyperparameters and enable simpler tuning at scale, we propose learning a Configuration-to-Performance Scaling Law (CPL): a mapping from the full training configuration to training performance. Because no simple functional form can express this mapping, we parameterize it with a large language model (LLM), and fit it with diverse open-source pretraining logs across multiple sources, yielding a Neural Configuration-to-Performance Scaling Law (NCPL). NCPL accurately predicts how training configurations influence the final pretraining loss, achieving 20-40% lower prediction error than the configuration-agnostic Chinchilla law and generalizing to runs using up to 10 more compute than any run in the training set. It further supports joint tuning of multiple hyperparameters with performance comparable to hyperparameter scaling law baselines. Finally, NCPL naturally and effectively extends to richer prediction targets such as loss-curve prediction, and can be adapted to new training settings by fine-tuning on only a small amount of additional logs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling law, pre-training, loss prediction, large language models, hyperparameters

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_osddOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

NCPL introduces the concept of fine-tuning LLMs for the task of predicting the performance of pretraining beyond the dimensions typically covered by scaling laws. Using the example of a Qwen3-1.7B model applied to two datasets showing the performance of training runs, NCPL demonstrates good accuracy compared to Chinchilla.

Critical strengths
  • NCPL generalizes scaling laws to new parametric dimensions and training configurations.
  • Prediction performance is better or comparable to other methods.
  • Well-written and easy-to-follow paper.
Critical weaknesses
  • The paper establishes that NCPL performs better than baselines, but does not explain how it works or how it arrives at those results.
  • Comparison against DL methods (TabM) and gradient boosting (CatBoost) needs further explanation.
  • Unclear meaning of “ground-truth optimal” in some evaluation benchmarks. [See below]
Questions for authors
  • You state that “hyperparameter selection without specifying an explicit functional form a priori by modeling the pretraining loss from heterogeneous training logs”. I don’t think the paper motivates that abandoning such functional forms is a good direction. Wouldn’t they be useful for understanding training behavior - in addition to performing prediction? Couldn’t your approach be used to discover new functional forms?

  • The evaluation clearly established performance in loss prediction, but the paper does not explain how NCPL actually works. Is the model discovering new scaling laws? Are old laws being adjusted to the provided dataset? What does the reasoning look like for out-of-distribution predictions? I think these are very relevant questions.

  • Scaling laws like Chinchilla are already known and included in the model’s training data. Is it possible that NCPL simply learns a simple modification of that law for the StepLaw dataset (Figure 6)? Did you investigate what functional form is used during prediction? Is it possible that the model simply applies minor adjustments to known scaling data to better fit into the provided dataset?

  • While reading Section 4.1, my immediate question was about the coverage of different parameters between the two datasets: are parameters sampled regularly and consistently? Section 4.4 seems to provide a partial answer - on the StepLaw dataset, which seems to vary fewer parameters, NCPL is frequently outperformed by other methods based on gradient boosting and deep learning. Wouldn’t this imply that NCPL is most usable for sparse datasets with diverse configurations? Wouldn’t a better approach be to employ hybrid inference, mixing more classical methods and NCPL depending on the parameters, configuration, and settings?

  • According to the Figure 6 caption, the plot shows relative losses of NCPL and power law against the “ground-truth optimal”. However, in a few plots, you report a negative relative loss. Wouldn’t this imply that the predicted configurations are better than “ground-truth optimal”? Can you please explain that?

3 comments
AuthorsOpenReview ↗

Responses to the Reviewer osdd

We thank the reviewer for recognizing NCPL’s ability to extend scaling laws to new dimensions and training configurations and its competitive predictive performance. Below, we address your concerns in detail.

W1/Q2: How NCPL works or how it arrives at those results.

We thank the reviewer for raising these questions. Understanding the mechanisms behind NCPL's predictions is an important direction. At its core, NCPL uses a Transformer-based neural network to learn configuration-to-performance mappings from a large collection of heterogeneous pretraining logs. Specifically, given the full training configuration, NCPL predicts a residual relative to the Chinchilla baseline. Thus, the Chinchilla prior provides the dominant scaling behavior with model and data sizes, while the Transformer learns configuration-dependent effects and their complex interactions.

As with many neural network-based approaches, its internal representations and prediction mechanisms are not directly interpretable. In particular, extracting an explicit symbolic scaling law or an interpretable computational circuit underlying its predictions is very challenging. What's more, a compact representation may not even exist for such a high-dimensional configuration space. We believe understanding these learned representations and mechanisms is an interesting direction for future work, which we leave for further investigation.

Q3: Scaling laws like Chinchilla are already known and included in the model’s training data. Is it possible that NCPL simply learns a simple modification of that law for the StepLaw dataset?

We do not think that NCPL's performance can be primarily attributed to memorizing or making minor modifications to Chinchilla's law. In particular, as discussed in Section 4.4 and Table 5, we also train NCPL from scratch, without relying on the pretrained knowledge of Qwen, and observe comparable performance on the StepLaw dataset. This suggests that the model can learn the configuration-to-performance mapping directly from pretraining logs.

Q1: Abandoning functional forms is not a good direction. They could be useful for understanding training behavior - in addition to performing prediction. NCPL could be used to discover new functional forms.

We agree that explicit functional forms are valuable for both prediction and understanding training behavior. However, existing scaling laws typically focus on one or two hyperparameters, rather than jointly modeling a broader range of training configurations. For example, Step Law jointly optimizes LR and BS while keeping other optimizer settings, including WD, fixed, and obtains the following optimal LR scaling law [1]:

LR∗=1.79N−0.713D0.307.LR^* = 1.79 N^{-0.713}D^{0.307}.

In contrast, MAI-Base-2 jointly optimizes LR and WD, obtaining a different functional form [3]:

-1.51 + 0.175\log TPP -0.573\log(\mathrm{steps}).$$ These examples illustrate that the resulting scaling relationships can depend on which hyperparameters are jointly optimized, highlighting the challenge of specifying a single functional form a priori to capture complex interactions among heterogeneous training configurations. Our goal is therefore not to abandon explicit functional forms, but to **avoid imposing potentially restrictive assumptions a priori when modeling these complex interactions**. We also agree NCPL can serve as a useful tool for discovering new functional forms. By analyzing the learned response surfaces and interactions, NCPL can help identify potentially interesting scaling relationships. We will clarify this perspective in the revised version. > Q4: NCPL is most usable for sparse datasets with diverse configurations? Hybrid inference? We agree the point that the reviewer raised. The two datasets indeed have substantially different configuration coverage. The StepLaw dataset conducts fine-grained sweeps over learning rate and batch size. In contrast, Marin explores a broader range of optimizers and hyperparameters through coordinate-descent sweeps with a sparse coverage. Indeed, our results show that NCPL is particularly advantageous for high-dimensional and heterogeneous configuration spaces with relatively sparse coverage, as in the Marin dataset. Such settings are common in LLM development, where many training factors, including optimizers, learning rates, batch sizes, and weight decay, jointly influence pretraining performance, while exhaustively exploring their combinations is prohibitively expensive. We also agree that hybrid inference is a promising direction. We view our work as a first step toward modeling heterogeneous configuration spaces, and leave the systematic design of hybrid predictors to future work.
AuthorsOpenReview ↗

Responses to the Reviewer osdd (continued)

Q5: the meaning of “ground-truth optimal”

We apologize for the confusion. For each (N,D)(N,D) setting, we define Lmin⁡L_{\min} as the minimum observed loss among the configurations evaluated in the StepLaw LR-BS grid search. For a configuration selected by NCPL or the power-law baseline, its loss is estimated by evaluating a quadratic surface fitted to the observed sweep results at the predicted (LR, BS). We then calculate the relative loss as L^−Lmin⁡Lmin⁡\frac{\hat L-L_{\min}}{L_{\min}}, where L^\hat L denotes the loss estimated by the quadratic fit.

Since the quadratic fit provides a smooth approximation to the discrete grid-search results, its predicted loss can occasionally be lower than the minimum observed loss Lmin⁡L_{\min}, resulting in a negative relative loss. We will clarify this evaluation procedure in the revised version.

Overall, we hope our response has addressed your concerns. We would also be happy to address any further questions during the discussion period.

Dear Reviewer osdd,

As the rebuttal period is nearing its end, we would be grateful if you could let us know whether you have any remaining concerns that we could address before it closes.

Thank you again for your time and effort in reviewing our work.

Best, Authors

Reviewer_CS2gOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes NCPL, which uses a fine-tuned language model to predict pretraining loss from training configurations. It improves loss prediction, helps select hyperparameters, and transfers to MoE models. However, experiments only cover models up to 1.2B parameters, so its ability to scale to larger models remains unclear.

Critical strengths
  1. Uses more training information. NCPL considers the full training configuration, capturing effects that traditional scaling laws may miss.
  2. Promising results. It predicts loss well and finds good hyperparameters in the tested settings.
Critical weaknesses
  1. Limited model scale. Experiments only cover models up to 1.2B parameters, leaving its effectiveness on much larger models unclear.
  2. Unclear advantage over simpler methods. Tabular models perform competitively on some benchmarks, so NCPL's advantage is not consistent.
Questions for authors
  1. Scaling to larger models: Have you tested NCPL on models at 7B parameters or larger? How well does it compare with traditional scaling laws?
  2. Separating model and data scaling: Since model size and token count change together in your OOD experiments, can you test them separately?
  3. Compared with simpler methods: Since XGBoost and other tabular models perform well on some benchmarks, when does NCPL actually work better?
  4. Hyperparameter selection: Can you verify NCPL's recommended configurations with independent training runs, especially when the best settings fall outside the tested range?
  5. MoE transfer: How well does NCPL perform using only the 179 MoE samples, without mixing in the original training data?
  6. Actual cost savings: After accounting for training NCPL and collecting calibration data, how many expensive training runs can it actually save?
3 comments
AuthorsOpenReview ↗

Responses to the Reviewer CS2g [1/2]

Thank you for your constructive review and for recognizing the strengths of NCPL of using more training information, and the promising results. Below, we address your concerns in detail.

W1,Q1: Limited model scale.

The limited training and evaluation range is primarily due to the lack of open-source pretraining logs at larger scales. In particular, existing open-source efforts typically release individual pretrained models, instead of the large collections of pretraining logs across diverse configurations, which are essential for training NCPL. We hope as the community releases more pretraining experiments covering broader model sizes and more diverse training setups, the range of model scales available for training and evaluation can be substantially expanded.

Recently, the Marin project released learning-rate sweep experiments for larger MoE models with up to 2.53B total parameters. We adapt NCPL to this setting following the same procedure as our MoE adaptation experiment in Section 4.3.

We evaluate on the largest 2.53B-parameter model and train on 154 sweep runs from smaller models. On the held-out largest model, NCPL achieves an OOD MAE of 0.020, substantially lower than the Chinchilla baseline’s 0.082. It also accurately tracks the U-shaped loss–learning-rate curve and predicts the ground-truth optimal learning rate for 4 of the 5 data scales. For the remaining scale, the predicted optimum incurs a ground-truth loss only 0.0097 above the true optimum.

W2, Q3: When does NCPL work better than simpler methods?

NCPL work better than tabular baselines in two ways:

  1. When multiple configurations vary jointly, NCPL demonstrates a clear advantage. The Marin dataset contains diverse training runs spanning different optimizers and hyperparameter settings. (By contrast, in the StepLaw dataset, only the training scale, learning rate, and batch size vary, while all other hyperparameters and the optimizer, are held fixed.) As shown in Figure 4, NCPL achieves the best performance on 7 out of 10 metrics on the Marin dataset and ranks a close second on the remaining three. No other baseline performs as consistently well.

  2. NCPL is able to adapt to new training settings. When adapting to new architectures or training environments, NCPL can leverage knowledge learned from large-scale pretraining logs and adapt using only a small number of additional runs. Our dense-to-MoE transfer experiment in Section 4.3 and Figure 9 demonstrates this capability.

Q2: Separating model and data scaling

In pretraining practice, model size and training-token count are often scaled together, following the compute-optimal scaling paradigm [1]. Our OOD experiments therefore evaluate and demonstrate NCPL's ability to extrapolate jointly along both dimensions. Moreover, the open-source pretraining logs we use (Marin and StepLaw) follow this joint scaling pattern, making it difficult to isolate the individual effects of the two dimensions.

[1]: Hoffmann et al. (2022), Training Compute-Optimal Large Language Models, NeurIPS 2022.

Q4: Verify NCPL's recommended configurations with independent training runs, especially when the best settings fall outside the tested range?

Due to computational constraints, we are unable to conduct independent pretraining runs. Instead, we evaluate NCPL's hyperparameter selection capability using existing open-source pretraining results. As shown in Section 4.2 (Figures 6 and 7), NCPL identifies near-optimal configurations in both ID and OOD settings. Importantly, the recommended configurations fall within the hyperparameter ranges covered by the collected pretraining logs.

Q5: How well does NCPL perform using only the 179 MoE samples, without mixing in the original training data?

We have already included this comparison in Figure 9 (labeled “NCPL-trained-solely-on-MoE”). Starting from the same backbone, training NCPL using only the 179 MoE samples performs significantly worse than pretraining on dense-model logs followed by fine-tuning on the MoE samples. Specifically, the hyperparameter configuration selected by the fine-tuned NCPL incurs a relative loss gap of only 0.11%, compared to over 5% for NCPL trained solely on the 179 MoE samples. This indicates that NCPL can transfer useful knowledge learned from large-scale pretraining logs and be efficiently adapted to new training settings with limited additional samples.

AuthorsOpenReview ↗

Responses to the Reviewer CS2g [2/2]

Q6: Actual cost savings

We first emphasize that training NCPL introduces relatively little additional overhead: its training cost is 44 H100 GPU-hours, substantially smaller than that of large-scale pretraining experiments. Moreover, NCPL can be trained using pretraining logs routinely collected during LLM development, without requiring additional experiments specifically for this purpose.

In practice, target-scale pretraining is often too expensive to repeat. The primary goal of scaling laws is therefore to identify near-optimal configurations before committing to expensive target-scale runs, rather than simply reducing the number of training runs.

A precise estimate of end-to-end compute savings is difficult, as it depends on the target training budget. Previous work on loss scaling laws [1, 2] and hyperparameter scaling laws [3] also does not provide a directly comparable end-to-end cost analysis accounting for these factors. Nevertheless, our hyperparameter selection results are comparable to those of Step Law [3] (Figure 6), demonstrating that NCPL can identify similarly near-optimal configurations. We leave a systematic evaluation of end-to-end compute savings to future work.

[1] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.

[2] Kaplan et al. (2020). Scaling Laws for Neural Language Models.

[3] Li et al. (2025). Predictable Scale: Part I — Optimal Hyperparameter Scaling Law in Large Language Model Pretraining.

Overall, we hope our response has addressed your concerns. If so, we would greatly appreciate it if you could reconsider your rating. We would also be happy to address any further questions during the discussion period.

Dear Reviewer CS2g,

As the rebuttal period is nearing its end, we would be grateful if you could let us know whether you have any remaining concerns that we could address before it closes.

Thank you again for your time and effort in reviewing our work.

Best, Authors

Reviewer_6B9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Neural Configuration-to-Performance Scaling Law (NCPL), a neural surrogate that predicts pretraining outcomes from the full training configuration, including model size, data size, architecture, optimizer, and hyperparameters. The authors fine-tune Qwen3-1.7B on more than 3,000 logs from Marin and StepLaw. They evaluate final-loss prediction, loss-curve prediction, hyperparameter selection, interactions between optimizer and weight decay, and few-shot adaptation from dense models to MoE settings. NCPL generally improves over Chinchilla-style and parametric baselines, although tabular baselines are competitive on the less heterogeneous StepLaw data.

Critical strengths
  1. The paper addresses a practically important problem: pretraining sweeps are expensive, and a predictor of configuration-dependent outcomes could reduce iteration cost. The input representation and experiments cover more configuration factors than classical scaling laws.

  2. The evaluation is broad within the available logs. It includes ID/OOD splits, loss curves, hyperparameter selection, optimizer interactions, tabular baselines, and MoE adaptation. The paper is also relatively candid about limitations and reports cases where the model overpredicts at the largest extrapolation scale.

Critical weaknesses
  1. The evidence does not yet establish a generally transferable scaling law. The OOD split is primarily based on model size while remaining within two source projects and related model/training distributions. Since model size, data size, architecture, and source are correlated, the results may reflect interpolation or project-specific surrogate modeling rather than a law that transfers across training systems.

  2. The main comparison is not sufficiently diagnostic of the benefit of a language-model backbone. On StepLaw, where only a small set of continuous factors varies, XGBoost/CatBoost/TabM and scratch-trained models are competitive. The paper should clarify what scientific or practical advantage NCPL provides over a standard tabular surrogate, beyond its capacity to accept heterogeneous serialized inputs.

Questions for authors
  1. Can the authors provide leave-one-project-out, leave-one-architecture-out, or leave-one-dataset-out evaluation? Without this, how should readers distinguish extrapolation from source/project memorization?

  2. What happens if the source identifier is removed or held out? How much of the reported OOD performance depends on this feature?

  3. Can the authors evaluate NCPL in a prospective sequential HPO setting, where only a small number of new runs may be queried and the next configuration is selected based on the predictor? The current experiments mainly select among already-swept configurations.

  4. Please report calibrated predictive intervals or uncertainty estimates. How should a practitioner detect when NCPL is extrapolating outside the support of the logs?

3 comments
AuthorsOpenReview ↗

Responses to the Reviewer 6B9B [1/2]

We thank the reviewer for recognizing the practical significance of our problem and the broad empirical evaluation. Below, we address the concerns in detail.

W1: The OOD split is based on model size; the source projects and related model/training distributions are fixed. Q1: Provide leave-one-project-out, leave-one-architecture-out, or leave-one-dataset-out evaluation?

We would like to clarify that we do not claim or expect NCPL to achieve zero-shot generalization to entirely new training setups, such as unseen architectures or data distributions. Such generalization would be unrealistic without any empirical experiments from the target setting. For example, reliably predicting the performance of a new architecture without running any experiments on it is highly challenging, even for an experienced ML researcher.

Instead, we demonstrate NCPL's three capabilities: (1) NCPL can make more accurate predictions based on diverse training configurations beyond NN and DD. This already make NCPL more useful in practice than classical scaling laws. (2) We evaluate NCPL's ability to extrapolate to larger model and data scales. This is also a common practice in frontier LLM development, where scaling ladders are constructed by training models of different sizes under a fixed training recipe to predict performance at larger scales (e.g., [1]). (3) efficient adaptation to new training settings using a small number of additional logs. Specifically, in Section 4.3 (Figure 9), NCPL trained on over 3,000 dense-model logs can be adapted to MoE using only 179 MoE samples, substantially outperforming training solely on these samples.

We will further clarify the scope of our generalization claims in the revised version.

[1] Microsoft AI (2026), Frontier Pre-training at Microsoft AI

W2: What scientific or practical advantage NCPL provides over a standard tabular surrogate?

NCPL work better than tabular baselines in two ways:

  1. When multiple configurations vary jointly, NCPL demonstrates a clear advantage. The Marin dataset contains diverse training runs spanning different optimizers and hyperparameter settings. (By contrast, in the StepLaw dataset, only the training scale, learning rate, and batch size vary, while all other hyperparameters and the optimizer, are held fixed.) As shown in Figure 4, NCPL achieves the best performance on 7 out of 10 metrics on the Marin dataset and ranks a close second on the remaining three. No other baseline performs as consistently well.

  2. NCPL is able to adapt to new training settings. When adapting to new architectures or training environments, NCPL can leverage knowledge learned from large-scale pretraining logs and adapt using only a small number of additional runs. Our dense-to-MoE transfer experiment in Section 4.3 and Figure 9 demonstrates this capability.

Q2: What happens if the source identifier is removed or held out? How much of the reported OOD performance depends on this feature?

We conduct an ablation by removing the source identifier from the input.

Marin

MethodID MAEID RMSEID SpearmanOOD MAEOOD RMSEOOD Spearman
NCPL(with source tag)0.01200.01840.97770.01980.02700.9446
NCPL(no source tag)0.01240.01890.97800.02910.03460.9409

Steplaw

MethodID MAEID RMSEID SpearmanOOD MAEOOD RMSEOOD Spearman
NCPL(with source tag)0.01030.01550.99580.02370.03390.9863
NCPL(no source tag)0.00950.01510.99600.02680.03480.9888

We ran the ablation that source identifier is removed. The performance does not drop significantly. We hypothesize that NCPL can infer which project a training run comes from based on its other configuration features. However, as discussed in our response to W1/Q1, we do not claim zero-shot generalization to entirely unseen projects.

AuthorsOpenReview ↗

Responses to the Reviewer 6B9B [2/2]

Q3: Evaluate NCPL in a prospective sequential HPO setting.

Thank you for this suggestion. We added an offline sequential HPO experiment using the four StepLaw OOD settings in Figure 6.

Each search begins with two random queries, followed by one configuration selected at a time, up to ten total queries. True losses remain hidden until the corresponding configurations are queried. After the initial two queries and after each subsequent query except the last, we update NCPL’s linear prediction head for ten gradient steps, using an equal mixture of queried configurations and historical training logs. The Transformer backbone remains frozen. The next configuration is selected by minimizing the predicted loss of the updated NCPL model.

We use ten paired random initializations per setting, yielding 40 search trajectories per method. The table reports mean relative loss above the best logged loss in each candidate pool. 0% indicates that the pool optimum has been found.

MethodAfter 3 queriesAfter 5 queriesAfter 10 queries
Random search0.8488%0.5391%0.2201%
Head-updated NCPL0.0914%0.0388%0.0000%

By the tenth query, head-updated NCPL finds the best logged configuration in all 40 search trajectories.

Our computational budget does not permit conducting additional pretraining experiments by ourselves, so we use existing StepLaw logs for this offline HPO experiment. Practitioners can apply the same select–run–update procedure online by executing the selected pretraining configurations, which may lead to more accurate hyperparameter selection.

Q4: Report calibrated predictive intervals or uncertainty estimates. How should a practitioner detect when NCPL is extrapolating outside the support of the logs?

We evaluated 90% and 80% split-conformal prediction intervals for NCPL’s final-loss predictions. We kept NCPL fixed and randomly split the original OOD logs into disjoint calibration and test sets. Neither set was used for NCPL training or checkpoint selection.

Using the held-out calibration data, we constructed symmetric prediction intervals around NCPL’s point predictions. The table below reports empirical coverage (the percentage of test runs whose observed final loss falls within the prediction interval) and mean interval width.

DatasetTarget coverageEmpirical coverageMean interval width
Marin90%88.9%0.0578
Marin80%77.9%0.0439
StepLaw90%88.8%0.1083
StepLaw80%80.4%0.0749

A practical diagnostic for extrapolation is to check whether configurations such as learning rate and batch size fall substantially outside the ranges covered by the training logs.

Overall, we hope our response has addressed your concerns. If so, we would greatly appreciate it if you could reconsider your rating. We would also be happy to address any further questions during the discussion period.

Dear Reviewer 6B9B,

As the rebuttal period is nearing its end, we would be grateful if you could let us know whether you have any remaining concerns that we could address before it closes.

Thank you again for your time and effort in reviewing our work.

Best, Authors

Reviewer_ZnWEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to incorporate hyperparameter configuration settings to improve the prediction of language model scaling. They propose a simple method which translates the configuration settings and their numerical values into a downstream model performance metric. This method essentially involves finetuning a small model (Qwen3-1.7B) to take as input the configuration setting names and values and predict the final pretraining loss for that particular training run. They show that incorporating the hyperparameter configuration can reduce error predicted by a Chinchilla scaling law. They also show it can be used to predict values on the loss curve for different optimizers, and also be adapted to scaling laws for different training settings (e.g. MoE architectures).

Critical strengths

This paper studies an important problem: how to reliably predict the performance of a model through the use of settings known at the start of training time. Scaling law research has important implications on how to improve and optimize training runs before they are committed.

Critical weaknesses

I am not thoroughly convinced that a language model is the best choice of model used to incorporate hyperparameter information. The results in table 6 and table 7 of the appendix suggest that a language model isn't so much better compared to some simpler tabular baselines, which to my understanding can only take as input the numerical inputs of the training settings. To put it another way, I am not sure that your NCPL method is able to fully leverage any predictive power in the qualitative text description of the training settings themselves. I would be interested to see if your trained regressor model would be able to predict performance of a qualitatively different training setup: for example, trying to predict performance/loss for an unseen optimizer. This could help determine whether the pretrained model contributes useful knowledge beyond fitting numerical relationships and categorical labels. Comparisons against tabular and randomly initialized neural baselines in these settings would strengthen the motivation for using an LLM regressor.

I think the headline result, which is compared against Chinchilla, needs a more careful interpretation. NCPL predicts a residual correction on top of a fitted Chinchilla scaling law. As a result, it feels a bit unfair to claim that NCPL can reduce prediction error than Chinchilla, because it receives additional configuration information that Chinchilla does not use. I would be interested to see if the tabular baselines in table 6 and 7 could also be used to predict without the residual. A matched comparison using the same inputs and residual targets would help distinguish the contributions of the scaling-law prior, configuration information, and pretrained language-model architecture.

Questions for authors
  • I am curious how the results may change with size, e.g. a bigger finetuned language model.
  • Do the tabular baselines also predict residuals relative to the same fitted baseline, or do they predict loss directly?
  • Since NCPL already receives model size NN and token count DD, could the authors discuss why direct loss prediction performs substantially worse than residual prediction, particularly OOD? Does the gap primarily reflect limited training-data coverage, difficulty learning power-law extrapolation, or interactions between scale and other configuration settings? This would help clarify which predictive behavior is learned by NCPL and which is supplied by the explicit Chinchilla prior.
4 comments
AuthorsOpenReview ↗

Responses to the Reviewer ZnWE [1/2]

Thank you for your constructive review and for recognizing the strong motivation and practical relevance of configuration-to-performance scaling laws. Below, we address your concerns in detail.

W2, Q2, Q3: Clarify the role of scaling-law prior, configuration information, and pretrained language-model architecture.

Q2: Do the tabular baselines also predict residuals relative to the same fitted baseline, or do they predict loss directly?

The tabular baselines in Table 4 also predict residuals with respect to the same fitted Chinchilla baseline, ensuring a fair comparison.

Q3: Why direct loss prediction performs substantially worse than residual prediction, particularly OOD? This would help clarify which predictive behavior is learned by NCPL and which is supplied by the explicit Chinchilla prior.

We attribute this gap primarily to the difficulty of transformer-based models in learning power-law extrapolation directly from data. We trained NCPL on synthetic data generated from simple power-law functions L(N,D)=A+BN−α+CD−βL(N,D)=A+BN^{-\alpha}+CD^{-\beta} as in the Chichilla law. Even in this controlled setting, NCPL cannot extrapolate reliably.

This suggests that the transformer backbone does not reliably learn the underlying power-law scaling behavior from data alone. Accordingly, the Chinchilla prior supplies the dominant power-law decrease in loss with model and data scales, while NCPL focuses on learning the more complicated effects of other training configurations and their interactions with model and data scales. We will further clarify in the revised version.

Thus, the dominant power-law decrease in loss with increasing model and data sizes is captured by the Chinchilla prior. The key advantage of NCPL, as we claim, lies in learning the more complicated interactions among training configurations across scales.

W2.1: Unfair to claim that NCPL can reduce prediction error than Chinchilla, because it receives additional configuration information that Chinchilla does not use.

We would like to clarify that the goal of such comparison is to demonstrate the value of incorporating full training configurations for more accurate performance prediction, which is itself an important contribution of our work.

To evaluate NCPL as a predictor given the same input information, we further compare it with tabular baselines, including XGBoost, CatBoost, and TabM. These comparisons demonstrate NCPL's advantages in high-dimensional and heterogeneous configuration spaces, such as Marin, where it achieves the best performance on 7 out of 10 metrics and ranks a close second on the remaining three (Section 4.4, Table 4). Furthermore, Section 4.3 demonstrates NCPL's ability to transfer knowledge from existing pretraining logs and adapt to new training settings using a small number of additional samples.

W2.2: Tabular baselines without the residual?

Consistent with the NCPL ablation reported in Table 3, our additional experiments show that directly predicting the absolute loss instead of the residual substantially weakens OOD generalization for the tabular baselines. All three baselines have worse OOD MAE, RMSE, and Spearman correlation on both datasets for final loss prediction. Although TabM retains relatively low absolute error for final loss prediction, its ranking accuracy (Spearman) deteriorates sharply on Marin OOD. This suggests a substantially weakened ability to capture the fine-grained effects of training configurations.

Each cell reports residual prediction → direct loss prediction.

Final loss prediction

OOD datasetBaselineMAE ↓RMSE ↓Spearman ↑
MarinXGBoost0.0325 → 0.18350.0375 → 0.18520.9227 → 0.9154
MarinCatBoost0.0345 → 0.09940.0379 → 0.10610.9501 → 0.9161
MarinTabM0.0186 → 0.02140.0235 → 0.02960.9456 → 0.7660
StepLawXGBoost0.0245 → 0.13770.0332 → 0.14490.9863 → 0.9391
StepLawCatBoost0.0229 → 0.07610.0310 → 0.09430.9883 → 0.8913
StepLawTabM0.0127 → 0.04630.0177 → 0.05240.9963 → 0.9704

Loss curve prediction

OOD datasetBaselineMAE ↓RMSE ↓
MarinXGBoost0.0385 → 0.13090.0667 → 0.1448
MarinCatBoost0.1208 → 0.13830.2546 → 0.1621
MarinTabM0.0892 → 0.25140.4760 → 0.9915
StepLawXGBoost0.0471 → 0.14830.1834 → 0.2190
StepLawCatBoost0.0385 → 0.07630.1709 → 0.1804
StepLawTabM0.0414 → 0.05410.1691 → 0.1722
AuthorsOpenReview ↗

Responses to the Reviewer ZnWE [2/2]

W2.3: Distinguish the contributions of the scaling-law prior, configuration information, and pretrained language-model architecture.

We agree that distinguishing the contributions of these three components is important. We clarify their respective roles below, and we'll clarify in our revised version.

  1. Configuration information: Incorporating full training configuration information enables more accurate performance prediction. As shown in Table 4, both NCPL and tabular baselines that leverage configuration information outperform the configuration-agnostic Chinchilla law. Moreover, it enables important applications such as joint hyperparameter selection (Section 4.3).

  2. Scaling-law prior: The Chinchilla prior captures the dominant power-law extrapolation trend with respect to model and data sizes, while NCPL (or other tabular methods) learn the more complicated effects of other training configurations and their interactions across scales.

  3. Pretrained language-model backbone: Compared with simpler tabular methods, the pretrained LM backbone offers two key advantages: (1) more accurate prediction in high-dimensional and heterogeneous configuration spaces, as demonstrated by our comparisons on Marin (Table 4); and (2) the ability to efficiently adapt to new training settings using only a small number of additional samples (Section 4.3).

Other concerns

W1: A language model isn't so much better? Is NCPL able to fully leverage any predictive power in the qualitative text description, e.g., predict performance for an unseen optimizer.

As discussed in our response to W2.3, the advantages we observe from the pretrained language-model backbone are more accurate prediction in high-dimensional and heterogeneous configuration spaces, and more efficient adaptation to new training settings.

To examine whether NCPL can leverage qualitative text description of training configurations, we conducted an additional experiment in which we augmented the training inputs with textual descriptions of optimizers, while holding out all training runs associated with one optimizer from the Marin dataset as the validation set. However, we did not observe a significant advantage of NCPL initialized from a pretrained language model compared to randomly initialization.

This suggests that our current prediction method, which directly maps a serialized training configuration to a scalar loss in a single forward pass, may not fully exploit the text understanding acquired during language-model pretraining. An interesting direction is to investigate whether incorporating CoT supervision into NCPL can encourage the model to explicitly interpret textual descriptions and reason about their implications for training performance.

Q1: how the results may change with size, e.g. a bigger finetuned language model.

We fine-tune NCPL using three different backbones: Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B. The results remain largely consistent across different backbone sizes, with no significant performance differences.

Marin

MethodID MAEID RMSEID SpearmanOOD MAEOOD RMSEOOD Spearman
Qwen3-0.6B0.01070.01700.97530.03070.03690.9422
Qwen3-1.7B0.01200.01840.97770.01980.02700.9446
Qwen3-4B0.01220.01790.97690.02700.03400.9440

Steplaw

MethodID MAEID RMSEID SpearmanOOD MAEOOD RMSEOOD Spearman
Qwen3-0.6B0.01090.01690.99470.02190.02970.9900
Qwen3-1.7B0.01030.01550.99580.02370.03390.9863
Qwen3-4B0.01020.01510.99600.02380.03250.9860

Overall, we hope our response has addressed your concerns. If so, we would greatly appreciate it if you could reconsider your rating. We would also be happy to address any further questions during the discussion period.

Dear Authors,

I very much appreciate the time and effort spent to address my concerns. I am still concerned about two experimental results:

  1. In Table 4, NCPL is not always the clear winner, so to me that limits the claim that a "neural ansatz" is required for capturing hyperparameter interactions. To support the paper's main claim, I am still hoping to see some cases or settings where NCPL clearly outperforms existing tabular baselines, or perhaps consider softening to a broader claim that modeling training configuration can be useful for better scaling law prediction.

  2. The additional experiments on qualititative text say there is no benefit to using a pretrained language model over a randomly initialized language model. This suggests to me that the design of a neural network architecture has not been fully explored to accurately and sufficiently model the hyperparameter configuration. For example, could you do away with language modeling entirely, and just use the 2-layer MLP to predict model performance from numerical values?

AuthorsOpenReview ↗

Response to Reviewer ZnWE

We thank the reviewer for these helpful suggestions. We agree that NCPL is not universally superior to simpler methods. On StepLaw, where configurations are systematically sampled over a relatively small number of hyperparameters, simpler methods remain competitive. Our claim is therefore that expressive neural predictors such as NCPL are particularly beneficial for high-dimensional and heterogeneous configuration spaces.

To further investigate whether a simpler neural architecture would suffice in such settings, we conducted additional experiments using plain MLPs with numerical configuration inputs.

Marin: ID/OOD Spearman correlation

MethodHidden layersWidthIDOOD
NCPL (Qwen3-1.7B)——0.9777 ± 0.00440.9446 ± 0.0084
MLP2640.9586 ± 0.00560.8101 ± 0.0551
MLP120480.9585 ± 0.01920.8797 ± 0.0155
MLP210240.9373 ± 0.04040.8671 ± 0.0184
MLP810240.8825 ± 0.00720.8800 ± 0.0034
MLP240960.9545 ± 0.02590.8793 ± 0.0209

All results are means and sample standard deviations over five independent random seeds. NCPL consistently achieves higher Spearman correlations than all tested MLP architectures. Notably, increasing the width and depth of the MLP does not close the performance gap, suggesting that a more expressive neural architecture, as adopted in NCPL, is beneficial for modeling such heterogeneous configurations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.