acceptodds
Withdrawn from ICLR 2027

When is predicting training loss worth the cost of initialization selection?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting among random initializations consumes computation that could instead train the selected network. We study this tradeoff under a fixed total compute budget using two predictors of future training loss: a separate frozen empirical neural tangent kernel (NTK) for each candidate, and a shared infinite-width limiting NTK used with each candidate’s own initial residual. For two-hidden-layer ReLU networks trained by full-batch gradient descent, the expected squared discrepancy between frozen-NTK predictions and actual outputs vanishes with width, uniformly over a fixed number of candidates and updates. We derive the limiting expected training loss after selection. Assume there are at least two candidates and the limiting initial-output covariance is positive definite. We compare selection by predicted future loss with selection by initial loss at the same training horizon. Future-loss selection yields a strictly lower limiting expected training loss if and only if the limiting dynamics scale residual directions by different absolute factors. Under a compute budget, this gain must also offset the updates forgone in scoring; we give an explicit sufficient condition in a realizable class with two error-decay rates. Extensive simulations illustrate the theoretical results and assess the tradeoff between prediction accuracy and computational cost..

Primary area:
learning theory
Keywords:
Neural tangent kernel, Random initialization, Initialization selection, Training-loss prediction, Computational budget

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CWcjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies when selecting a neural network initialization based on predicted future training loss is worth its computational cost under a fixed compute budget. The authors consider two NTK-based predictors: a candidate-specific empirical NTK and a shared infinite-width limiting NTK. They establish that frozen empirical-NTK predictions approximate actual training dynamics as network width increases, and characterize when future-loss selection achieves lower expected training loss than initial-loss selection. They further derive a sufficient condition under which the selection gain outweighs the cost of fewer training updates. Experiments on synthetic regression tasks and a small real-world regression dataset compare the proposed approaches with random selection, initial-loss selection, brief training trials, and successive halving. The results show that the effectiveness of initialization selection depends on the compute budget, and that more accurate predictions do not necessarily lead to better final performance.

Critical strengths

The paper provides a theoretical characterization of when selecting an initialization based on predicted future loss improves upon initial-loss selection. In particular, Proposition 2 gives a necessary and sufficient condition for a strict improvement at a common training horizon, while Proposition 3 provides a sufficient condition for the selection gain to outweigh the additional computational cost. The paper accounts for the computational costs of candidate generation, scoring, and training when comparing selection strategies. The distinction between prediction accuracy and final performance under an equal compute budget is particularly interesting. The finding that the less expensive limiting NTK can outperform the more accurate empirical NTK provides a useful insight into the trade-off between prediction quality and computational efficiency.

Critical weaknesses

The proposed approach requires constructing NTKs to predict future training loss, which can introduce substantial computational overhead, particularly for the candidate-specific empirical NTK. Although the authors account for these costs in their budget comparisons, the experiments also show that the cheaper limiting NTK often achieves better final performance despite less accurate predictions. This raises the question of whether computing candidate-specific NTKs is justified by the additional selection benefit. Given that computational efficiency is central to the paper, a clearer understanding of when the cost of NTK computation is worthwhile would strengthen the contribution.

The experiments are primarily conducted on small synthetic regression tasks, using only 32 or 64 training samples and relatively small two-hidden-layer ReLU networks. The only real-world evaluation uses the Diabetes dataset with 128 training samples. Moreover, the main experiments consider relatively short training budgets of 20 or 50 updates. While these experiments help illustrate the theoretical findings, it remains unclear whether the observed benefits generalize to larger datasets, deeper architectures, and longer training runs. This limits the evidence for the practical applicability of the proposed approach.

Questions for authors

Given the potentially high computational cost of constructing empirical NTKs, could the authors further clarify when this additional cost is justified? In particular, the experiments show that limiting NTK often achieves better final performance under the same compute budget despite being less accurate. Under what conditions would computing a separate empirical NTK for each candidate provide sufficient benefit to outweigh its additional cost? Could the authors also compare against less expensive predictors, such as initial gradient-based scores?

Could the authors extend the experiments to larger datasets and larger models? The current evaluation is largely restricted to small regression problems, making it difficult to assess whether the proposed initialization selection strategy remains effective in more realistic learning settings. It would be particularly helpful to evaluate whether the improvements over initial-loss selection and training-based baselines persist at a larger scale.

Could the authors investigate whether the benefits of future-loss selection persist over substantially longer training horizons? The main experiments use budgets corresponding to only 20 or 50 training updates. It is unclear whether the initial ranking advantages remain meaningful after longer training, or whether changes in the NTK reduce the accuracy of future-loss predictions.

Reviewer_6sAJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether, under a fixed architecture, optimizer, and total compute budget, it is worthwhile to select among multiple random initializations by predicting their future training loss. It compares two predictors:

  • Empirical NTK: freeze a separate empirical neural tangent kernel at each candidate’s initialization and use it to predict that candidate’s future training loss.
  • Limiting NTK: use the same deterministic infinite-width kernel for every candidate, while retaining each candidate’s own initial residual. This is cheaper, but ignores finite-width differences between candidates’ kernels.

Theoretically, the authors prove that, for fixed data, candidate count, and training horizon, predictions based on the frozen empirical NTK converge to the actual training dynamics as network width grows. They then use the Gaussian-process limit to characterize expected loss after selection. Their clearest result is that, when the limiting initial-output covariance is full rank and there are at least two candidates, selecting by future loss strictly improves on selecting by initial loss if and only if training scales different residual directions by unequal absolute factors. In other words, if training scales all directions equally, ranking candidates by predicted future loss does not improve the selection. This is a clean condition, but its geometric intuition is not especially surprising: future-loss ranking can help when training affects error directions differently enough to change the candidates’ ranking. The contribution is primarily to turn this intuition into an exact characterization and analyze whether the selection gain can offset the training updates lost to scoring, rather than to introduce a wholly unfamiliar mechanism.

The paper does not demonstrate that its approach is effective in practical neural-network settings. The experiments are limited to small, two-layer fully connected networks trained on low-dimensional regression tasks, so it remains unclear whether the findings extend to larger or deeper networks and commonly used adaptive optimizers.

Critical strengths
  • This is a technically solid paper, with detailed proofs covering kernel convergence, gate changes, the empirical-NTK approximation, singular covariance cases, and expected loss after selection.
  • The theoretical derivation is interesting and yields clean, intuitively interpretable results.
  • The experiments are solid for a theory paper. Each of the two main regression settings uses 200 datasets and 128 candidate groups per dataset. The authors also compare candidate counts, budgets, network widths, both NTK predictors, short training trials, and successive halving.
Critical weaknesses
  • The equal-budget results depend on measured implementation costs. The paper provides the cost model, measurement procedure, and resulting update allocations, but the PDF does not report the hardware, software environment, or raw operation costs in enough detail to assess how robust the results would be under a different implementation or on different hardware.

  • The practical evidence is limited in scope. Most results come from low-dimensional synthetic regression, small training sets, and two-layer fully connected networks. The real-data experiments cover only one relatively small regression dataset.

  • The technical organization and claims are generally clear, but the presentation could better accommodate readers unfamiliar with NTK research. The introduction uses “empirical NTK” and “limiting NTK” without first explaining what an NTK is or why it can predict future training loss. The first explicit formula definition in the main text, K0=J0J0⊤K_0=J_0J_0^\top, appears immediately after Eq. (2); the supplement restates and expands the definition, but does not provide the intuitive explanation missing from the introduction. Readers are left to infer how this expression relates to training dynamics before they can understand the frozen-NTK prediction.

  • The authors claim that, under a fixed compute budget, more accurate training-loss predictions need not produce lower final loss, and that prediction accuracy and scoring cost jointly affect initialization selection. This is an interesting empirical observation. However, the connection between this observation and the theory is not sufficiently explained. The theoretical results characterize training loss under specific residual-scaling conditions, whereas the experiments primarily report test loss; the paper does not clearly show that its experimental settings instantiate the theoretical conditions. Experiments do not necessarily need to be derived from the theory, but the authors should explain whether these experiments test the theoretical regime or serve as a separate empirical validation. If they are intended to support a unified theoretical and empirical contribution, the paper should make that bridge explicit.

Questions for authors
  1. How do the experimental settings relate to the sufficient conditions in Proposition 3? Do any of the reported finite-width experiments instantiate those conditions, and how should the training-loss theory be connected to the test-loss comparisons?

  2. Could you report the hardware, software environment, and measured primitive operation costs, and assess whether the equal-budget rankings remain stable under plausible changes in those costs?

  3. Could you introduce the intuition behind the NTK before using it in the introduction—for example, how it connects output sensitivities to training dynamics—and explain why this makes initialization selection an important problem?

Reviewer_dEsfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether selecting an initialization by predicted training loss is worth the computation spent on selection. The authors derive conditions for lower expected training loss after selection in the infinite-width limit and illustrates the trade-off on small regression problems.

Critical strengths
  • The budget accounting highlights the importance of computational costs, and is reminiscent of recent work on compute-optimal scaling laws. Within the setup considered, the potential sources of computational costs seems to be thorough.
  • The problem of initialization selection in the manner described in the paper has some compelling ideas. For instance, the observation that there can be substantial variation within different realizations is interesting. The paper could be strengthened further by focusing on establishing this before motivating their method.
Critical weaknesses
  • I am not convinced of the practical motivation. The authors fix a number of important contributors to performance (such as the architecture and optimizer) then select initializations. Although Section 6 cites prior work on initializations, much of those are in designing distributions with particular properties or meta-learning methods. The act of choosing an initialization is reminiscent of practice in deep RL, where initialization can have large effects on performance. The submission does not obviously establish that initialization selection addresses a practically important bottleneck, whether it be at small scale or large scale.
  • The paper and theory are very unclear. The introduction uses a number of uncommon terms that are only defined later, meaning that the abstract and introduction does nothing but confuse. The notation and propositions become very dense very quickly. Despite claiming to build on previous work, the paper does not make clear what is added beyond the existing NTK literature. I am also confused because a number of more recent papers have begun to study "feature learning regimes" outside of NTK, claiming that NTK dynamics can fail to capture the way that deep networks learn features. It isn't clear how useful these predictions will be for deep networks where feature learning becomes important.
  • The experiments are extremely small in scale. If you are using synthetic regression as a problem, then you are able to generate as much data as needed. Why are there only 32 training examples? While the supplement adds more experiments, they are similarly small in scale. My original thinking when I started to read this paper was that the problem of predicting loss from a given parameter initialization is challenging; these experiments do not give me confidence that such prediction is possible beyond these restricted settings.
Questions for authors
  • Beyond the small regression settings studied, where would selecting among random initializations be useful with all other training choices fixed? Do you anticipate that this is even possible in settings where training is computationally demanding?
  • How much variation in the loss is due to the initialization selected. It would be good to present this as a function of the number of updates. how much of that variation can your selection method exploit? Does any advantage persist with further training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.