acceptodds
Under review as a conference paper at ICLR 2027

When Does Scale Trade for Time?

Abstract

Can a larger model trained briefly behave like a smaller model trained for longer? We study this question, which we call scale-time equivalence, in a deliberately small and controlled setting where it can be derived, measured, and falsified. Theoretically, we analyze a random subspace model, in which the trained network occupies a random -dimensional subspace of a larger nonlinear model. This model contains Gaussian linear random-feature models, a workhorse of scaling theory, as the special case of a quadratic loss, but, unlike them, allows the network output to depend nonlinearly on the trainable parameters. We prove that under fixed-learning-rate gradient flow, the function learned by a model of scale after time is approximated by a -independent trajectory evaluated at , with an error that decays as for any fixed value of . The bound grows exponentially only in the effective training progress , not along the scale-time tradeoff itself, so it guarantees equivalence between models of very different scales as long as both are large and their shared value of is moderate. Empirically, in MLPs and CNNs trained with fixed-learning-rate SGD on MNIST, SVHN, and CIFAR-10, the number of epochs needed to generalize follows an approximate power law in model scale, with a common exponent across datasets, architectures, and training-set sizes, and small models trained for long predict large models trained briefly. With Adam, which lies outside the theorem's assumptions, the equivalence visibly breaks. Finally, we conjecture, and give preliminary evidence, that where scale-time equivalence holds, parameter-wise and time-wise double descent share a mechanism: the relative rates at which signal and noise are acquired.

Primary area:
learning theory
Keywords:
scaling law, scaling, parameter count, training time, model size

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6hkPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the potential trade-off between model scale and training time under a fixed training dataset, referred to as scale-time equivalence. Specifically, it examines whether a larger model trained for a shorter duration can achieve performance comparable to that of a smaller model trained for a longer duration.

On the theoretical side, the authors introduce a random subspace model in which the trainable parameters are restricted to a random pp-dimensional subspace of a larger parameter space. They prove that, under gradient flow with a fixed learning rate, the model's representation can be approximated by a model-size-independent reference trajectory evaluated at the rescaled time ptpt. Furthermore, when ptpt is held fixed, the approximation error decreases at a rate of O(p−1/2)\mathcal{O}(p^{-1/2}) as pp increases.

On the empirical side, the authors validate this phenomenon using MLPs and CNNs on the MNIST, SVHN, and CIFAR-10 datasets. The results suggest that, under SGD with a fixed learning rate, the training time required to reach a specified generalization-error threshold is approximately inversely proportional to an empirically defined effective model size. Moreover, the training trajectories of smaller models can, to some extent, predict the behavior of larger models. In contrast, this equivalence becomes substantially weaker or breaks down under the Adam optimizer. The authors also attempt to provide a unified mechanistic explanation for parameter-wise double descent and time-wise double descent through scale-time equivalence.

Critical strengths

1. The research question is theoretically meaningful.

The paper explores the intrinsic relationship between model scale and training time, offering a somewhat novel perspective compared with conventional scaling-law studies that primarily focus on the relationships among model size, dataset size, and final performance. Understanding this relationship could provide useful insights into optimization dynamics across different model scales and how changes in model size affect the training process.

2. The theoretical analysis provides a clear mechanistic explanation.

The authors investigate this phenomenon within an analytically tractable random subspace model. In this setting, the expectation of the effective optimization matrix KRR⊤K⊤KRR^\top K^\top in representation space grows linearly with the model size pp, while its random fluctuations scale as O(p)\mathcal{O}(\sqrt{p}). Building on this observation, the authors establish an approximate correspondence between model scale and training time through the product ptpt, together with a finite-size approximation error bound. This analysis provides a clear mathematical explanation for scale-time equivalence.

Critical weaknesses

The theoretical guarantee is limited in its range of effective training times, and its implications for practical optimization remain insufficiently clarified. This is my primary concern about the paper.

Although Theorem 3.1 establishes that, as p→∞p \to \infty with s=pts=pt held fixed, the model's representation approaches the reference trajectory with an approximation error of O(p−1/2)\mathcal{O}(p^{-1/2}), the error bound in Equation (4) contains the exponential term $ \exp(\eta h\lambda_K pt)-1. $ Consequently, for any fixed and finite model size, the bound can rapidly become loose as the effective training time ptpt increases, eventually ceasing to provide a meaningful approximation guarantee.

I understand that the theorem is intended to establish a local form of scale-time equivalence rather than a uniform guarantee over the entire training trajectory. However, the paper does not sufficiently clarify how large the effective training-time range covered by this theoretical guarantee is for practically attainable model sizes, or whether this range encompasses optimization or generalization phases of practical relevance.

I believe the authors should more clearly explain the scope and significance of this local theoretical result and further investigate how it relates to different stages of optimization in practical training.

Questions for authors

1. On the significance of local scale-time equivalence.

Theorem 3.1 primarily establishes an approximate equivalence in the regime of large pp and bounded ptpt. Could the authors further clarify how this theoretically justified regime relates to different stages of practical optimization, including the onset of generalization, the phase of rapid error reduction, and the near-convergence regime?

I believe clarifying this point would help readers better understand the practical significance of the theoretical results.

2. On the effective model size in practical neural networks.

Could the authors provide further intuition or justification for why the effective model size across different network architectures can be approximately characterized by the cube root of the number of parameters? Furthermore, how robust is this relationship to different parameter initialization schemes and normalization choices?

Overall Assessment

Overall, I find the research question compelling and the proposed theoretical mechanism insightful. However, I would like to see a more thorough clarification of the connection between the local theoretical guarantee and the behavior of practical optimization processes. If my primary concern can be convincingly addressed or resolved, I would be willing to revise my evaluation of the paper upward accordingly.

Reviewer_4Fn5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors aim to understand whether large models behave like small models under certain conditions, exploring a relationship between model size and training time / performance. They report that MLPs and CNNs trained with SGD on small vision benchmarks show a power-law tradeoff between scale and the time needed to generalise. They also conjecture a shared mechanism for parameter-wise and time-wise double descent.

Critical strengths
  • The paper is very clear about what is proved, observed and conjectured.
  • The attempt to explain parameter-wise and time-wise double descent with a single mechanism is admirable, and it comes with discriminating tests.
Critical weaknesses
  • In the random subspace model, scale p is equivalent to a learning rate of ηp, and the effect vanishes under NTK-style normalisation. The result therefore reduces to “larger models have a larger effective learning rate” and does not isolate a mechanism by which larger networks trained briefly behave like smaller ones trained longer. It is also not connected to the experiments: the theorem is not applied to the trained networks, and the 1/3 exponent that yields the slope of −1 is fitted, not predicted. The observed tradeoff could equally be explained by the greater capacity of larger networks, which would let them reach the threshold sooner without following the same trajectory. It may also hold only for the narrow range tested.
  • The double descent evidence isn't very strong. It also starts using an ADAM optimiser, leaving behind the assumptions made at the beginning.
Questions for authors
  • Does the speed-up disappear under muP or NTK parameterisation, as the theory predicts?
  • Does the equivalence survive in discrete time near the stability limit?
  • Can the double-descent conjecture be tested with SGD, or in the random-subspace model itself?
  • What do the authors believe this work adds to our understanding of network dynamics?
Reviewer_Shj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes a random subspace model and shows that a larger model acts similar to a smaller model trained for more steps under some conditions, at a fixed learning rate. The paper provides experiments showing this equivalence holds for predicting number of epochs to generalize for image models trained with SGD if the model size is discounted by an exponent of 1/3.

Critical strengths

Predicting a large model trained briefly from a small model’s training is a highly valuable goal and this work analyzes theoretically and empirically whether that is possible.

Critical weaknesses

I'm afraid the claimed equivalence between a pp times larger model and a model trained for pp times longer is entirely an artifact of the fixed learning rate. It is well known that learning rates do not share the same unit across model scales. In the particular model this paper analyzes, fixing the learning rate the pp times bigger model simply moves pp times faster in expectation in function space as the authors show in Section 3.2. This choice of fixed learning rate, known as Standard Parameterization, is the wrong parameterization for learning rate across scales, whereas the NTK Parameterization and Maximal Update Parameterization would be preferred in the kernel and feature-learning regime respectively and lead to consistent learning dynamics across scales at fixed steps, directly contradicting the paper's claim. On L248, the authors acknowledge that normalization would completely remove this effect, yet such normalization is commonly used and baked into the NTK and Maximal Update Parameterizations for exactly the purpose of ensuring the same value of learning rate means the same rate of update in function space across scales.

Yang and Hu. "Tensor programs iv: Feature learning in infinite-width neural networks." International Conference on Machine Learning. PMLR, 2021.

Questions for authors

Instead of using a fixed learning rate, do you see observe the scale-time equivalence if the learning rate is tuned at each scale? That would be the relevant question as the equivalence would not be meaningful if the 10x larger model simply had a 10x smaller learning rate compared to what is optimal.

Comments on the paper

1 comment

Dear Reviewers,

Thank you to those who have already engaged with the authors' rebuttal. As we approach the end of the discussion period, if you have not yet replied, please do so soon.

Thank you for your time and effort in contributing to the ICLR community.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.