- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a framework for predicting training trajectories across model scales by combining endpoint scaling laws with a shared normalized trajectory. It uses convex-optimization bounds to motivate learning-rate and weight-decay scaling, introduces a fitted training-progress offset, and proposes a parameterized trajectory. The framework is extended empirically to evaluation metrics, parameter norms, and multimodal models.
Critical strengths
Predicting complete training trajectories from smaller-scale experiments is a relevant problem. The paper includes experiments across several architectures, optimizers, and observables. The held-out forecast for the 7B model in Figure 4 is potentially useful.
Critical weaknesses
-
The convex-optimization analysis does not establish the proposed scaling law.
Theorem 1 provides a convergence upper bound under convexity and boundedness assumptions. The basic rate is standard. Such an upper bound does not establish the actual loss asymptotics, its tightness, or a common normalized trajectory across model sizes. Moreover, the quantities appearing in the bound depend on model size, and their scaling is not established. The transfer from convex gradient descent to nonconvex neural networks trained with Muon or AdamW also lacks a sufficient theoretical justification. Consequently, the theorem provides limited support for the central explanation of collapse.
-
The “any-time-any-scale” claims are substantially broader than the demonstrated scope.
The proposed transformation introduces a burn-in offset, while the experimental recipe requires sufficiently large models or training budgets and fixes several aspects of the training configuration. These restrictions should be reflected in the claims. The paper needs a clearer definition of universality, including the admissible model families, training regimes, observables, and approximation errors.
-
The assumed separation of training progress and model scale is insufficiently justified.
The final law represents reducible loss as a compute-dependent amplitude multiplied by a common trajectory . I recognize that this is formulated within a fixed training recipe and that the trajectory can depend on the learning-rate schedule. Nevertheless, the effects of batch size, optimizer, learning rate, regularization, and data distribution are largely implicit in the recipe or fitted coefficients. The paper does not establish when these effects preserve the proposed separation or when scale-dependent changes in trajectory shape become important. Fitting a common is insufficient to establish its predictive validity across configurations.
-
The reported predictive accuracy requires more reproducible evidence.
Figure 4 reports a validation-loss prediction error below 0.5%, but the complete fitting and evaluation pipeline is not sufficiently transparent for independent verification. Please provide the training and fitting code, numerical learning curves, parameter constraints, preprocessing and interpolation procedures, and exact fitting/evaluation splits. In particular, it should be possible to verify that no information from the target model’s trajectory or endpoint enters any stage of fitting or model selection.
-
The multimodal extension lacks a corresponding analysis of joint training dynamics.
Section 6 specifies a weighted combination of text and image losses and presents multimodal experiments. However, the theory does not characterize interactions between these objectives, their relative weights, modality mixtures, or shared parameters. Applying a similar fitted functional form to language and multimodal models provides empirical evidence, but does not establish a common optimization mechanism. The claimed unification needs a more precise meaning and stronger justification.
-
The validation protocol is insufficient for the breadth of the universality claim.
Although the paper includes multiple experimental configurations, much of the evidence consists of normalized curves fitted within each configuration. The clearest held-out forecasting example is the 7B prediction in Figure 4. A broader claim requires systematic held-out-scale and cross-configuration evaluations, comparisons with existing collapse methods and simpler trajectory models, and uncertainty estimates. Ablations should also isolate the contributions of the proposed hyperparameter scaling, time offset, and closed-form trajectory.
Questions for authors
-
What additional assumptions make the convex upper bound a tight description of neural-network loss, and what establishes the required relationships between its model-dependent coefficients across scales?
-
Which components of the framework are shared across configurations, and which must be refitted? Can the authors specify and experimentally test the boundaries of the claimed universality?
-
Does the proposed closed-form trajectory improve held-out prediction over directly estimating from reference curves? What are the separate contributions of the time offset and hyperparameter-scaling rules?
-
How does the framework account for modality mixtures, loss weights, and interactions between text and image objectives?
-
Will the authors provide code and numerical training trajectories sufficient to reproduce the reported forecasts and compare them against relevant baselines?