- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the potential trade-off between model scale and training time under a fixed training dataset, referred to as scale-time equivalence. Specifically, it examines whether a larger model trained for a shorter duration can achieve performance comparable to that of a smaller model trained for a longer duration.
On the theoretical side, the authors introduce a random subspace model in which the trainable parameters are restricted to a random -dimensional subspace of a larger parameter space. They prove that, under gradient flow with a fixed learning rate, the model's representation can be approximated by a model-size-independent reference trajectory evaluated at the rescaled time . Furthermore, when is held fixed, the approximation error decreases at a rate of as increases.
On the empirical side, the authors validate this phenomenon using MLPs and CNNs on the MNIST, SVHN, and CIFAR-10 datasets. The results suggest that, under SGD with a fixed learning rate, the training time required to reach a specified generalization-error threshold is approximately inversely proportional to an empirically defined effective model size. Moreover, the training trajectories of smaller models can, to some extent, predict the behavior of larger models. In contrast, this equivalence becomes substantially weaker or breaks down under the Adam optimizer. The authors also attempt to provide a unified mechanistic explanation for parameter-wise double descent and time-wise double descent through scale-time equivalence.
Critical strengths
1. The research question is theoretically meaningful.
The paper explores the intrinsic relationship between model scale and training time, offering a somewhat novel perspective compared with conventional scaling-law studies that primarily focus on the relationships among model size, dataset size, and final performance. Understanding this relationship could provide useful insights into optimization dynamics across different model scales and how changes in model size affect the training process.
2. The theoretical analysis provides a clear mechanistic explanation.
The authors investigate this phenomenon within an analytically tractable random subspace model. In this setting, the expectation of the effective optimization matrix in representation space grows linearly with the model size , while its random fluctuations scale as . Building on this observation, the authors establish an approximate correspondence between model scale and training time through the product , together with a finite-size approximation error bound. This analysis provides a clear mathematical explanation for scale-time equivalence.
Critical weaknesses
The theoretical guarantee is limited in its range of effective training times, and its implications for practical optimization remain insufficiently clarified. This is my primary concern about the paper.
Although Theorem 3.1 establishes that, as with held fixed, the model's representation approaches the reference trajectory with an approximation error of , the error bound in Equation (4) contains the exponential term $ \exp(\eta h\lambda_K pt)-1. $ Consequently, for any fixed and finite model size, the bound can rapidly become loose as the effective training time increases, eventually ceasing to provide a meaningful approximation guarantee.
I understand that the theorem is intended to establish a local form of scale-time equivalence rather than a uniform guarantee over the entire training trajectory. However, the paper does not sufficiently clarify how large the effective training-time range covered by this theoretical guarantee is for practically attainable model sizes, or whether this range encompasses optimization or generalization phases of practical relevance.
I believe the authors should more clearly explain the scope and significance of this local theoretical result and further investigate how it relates to different stages of optimization in practical training.
Questions for authors
1. On the significance of local scale-time equivalence.
Theorem 3.1 primarily establishes an approximate equivalence in the regime of large and bounded . Could the authors further clarify how this theoretically justified regime relates to different stages of practical optimization, including the onset of generalization, the phase of rapid error reduction, and the near-convergence regime?
I believe clarifying this point would help readers better understand the practical significance of the theoretical results.
2. On the effective model size in practical neural networks.
Could the authors provide further intuition or justification for why the effective model size across different network architectures can be approximately characterized by the cube root of the number of parameters? Furthermore, how robust is this relationship to different parameter initialization schemes and normalization choices?
Overall Assessment
Overall, I find the research question compelling and the proposed theoretical mechanism insightful. However, I would like to see a more thorough clarification of the connection between the local theoretical guarantee and the behavior of practical optimization processes. If my primary concern can be convincingly addressed or resolved, I would be willing to revise my evaluation of the paper upward accordingly.
Dear Reviewers,
Thank you to those who have already engaged with the authors' rebuttal. As we approach the end of the discussion period, if you have not yet replied, please do so soon.
Thank you for your time and effort in contributing to the ICLR community.
Best,
AC