- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies when selecting a neural network initialization based on predicted future training loss is worth its computational cost under a fixed compute budget. The authors consider two NTK-based predictors: a candidate-specific empirical NTK and a shared infinite-width limiting NTK. They establish that frozen empirical-NTK predictions approximate actual training dynamics as network width increases, and characterize when future-loss selection achieves lower expected training loss than initial-loss selection. They further derive a sufficient condition under which the selection gain outweighs the cost of fewer training updates. Experiments on synthetic regression tasks and a small real-world regression dataset compare the proposed approaches with random selection, initial-loss selection, brief training trials, and successive halving. The results show that the effectiveness of initialization selection depends on the compute budget, and that more accurate predictions do not necessarily lead to better final performance.
Critical strengths
The paper provides a theoretical characterization of when selecting an initialization based on predicted future loss improves upon initial-loss selection. In particular, Proposition 2 gives a necessary and sufficient condition for a strict improvement at a common training horizon, while Proposition 3 provides a sufficient condition for the selection gain to outweigh the additional computational cost. The paper accounts for the computational costs of candidate generation, scoring, and training when comparing selection strategies. The distinction between prediction accuracy and final performance under an equal compute budget is particularly interesting. The finding that the less expensive limiting NTK can outperform the more accurate empirical NTK provides a useful insight into the trade-off between prediction quality and computational efficiency.
Critical weaknesses
The proposed approach requires constructing NTKs to predict future training loss, which can introduce substantial computational overhead, particularly for the candidate-specific empirical NTK. Although the authors account for these costs in their budget comparisons, the experiments also show that the cheaper limiting NTK often achieves better final performance despite less accurate predictions. This raises the question of whether computing candidate-specific NTKs is justified by the additional selection benefit. Given that computational efficiency is central to the paper, a clearer understanding of when the cost of NTK computation is worthwhile would strengthen the contribution.
The experiments are primarily conducted on small synthetic regression tasks, using only 32 or 64 training samples and relatively small two-hidden-layer ReLU networks. The only real-world evaluation uses the Diabetes dataset with 128 training samples. Moreover, the main experiments consider relatively short training budgets of 20 or 50 updates. While these experiments help illustrate the theoretical findings, it remains unclear whether the observed benefits generalize to larger datasets, deeper architectures, and longer training runs. This limits the evidence for the practical applicability of the proposed approach.
Questions for authors
Given the potentially high computational cost of constructing empirical NTKs, could the authors further clarify when this additional cost is justified? In particular, the experiments show that limiting NTK often achieves better final performance under the same compute budget despite being less accurate. Under what conditions would computing a separate empirical NTK for each candidate provide sufficient benefit to outweigh its additional cost? Could the authors also compare against less expensive predictors, such as initial gradient-based scores?
Could the authors extend the experiments to larger datasets and larger models? The current evaluation is largely restricted to small regression problems, making it difficult to assess whether the proposed initialization selection strategy remains effective in more realistic learning settings. It would be particularly helpful to evaluate whether the improvements over initial-loss selection and training-based baselines persist at a larger scale.
Could the authors investigate whether the benefits of future-loss selection persist over substantially longer training horizons? The main experiments use budgets corresponding to only 20 or 50 training updates. It is unclear whether the initial ranking advantages remain meaningful after longer training, or whether changes in the NTK reduce the accuracy of future-loss predictions.