acceptodds
Withdrawn from ICLR 2027

The art and principle of universal scaling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling laws have become the bedrock for foundation model training, yet standard scaling laws are largely restricted to evaluating cross-entropy loss at terminal checkpoints – leaving the continuous dynamics of general observables under-explored. In this work, we establish universal scaling laws across large language models (LLM) and multi-modal models (MMLM) that accurately characterize the any-time-any-scale trajectories of losses, evaluation metrics, and parameter norms. To establish this unified framework, we integrate three key pillars: conventional end-point scaling laws, convex optimization theory, and an enhanced collapse transformation. We systematically validate this framework across diverse model architectures, optimizers, training hyperparameters, compute regimes, and modalities, providing a principled recipe for universal scaling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
convex optimization, universal scaling law, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_py1hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework for predicting training trajectories across model scales by combining endpoint scaling laws with a shared normalized trajectory. It uses convex-optimization bounds to motivate learning-rate and weight-decay scaling, introduces a fitted training-progress offset, and proposes a parameterized trajectory. The framework is extended empirically to evaluation metrics, parameter norms, and multimodal models.

Critical strengths

Predicting complete training trajectories from smaller-scale experiments is a relevant problem. The paper includes experiments across several architectures, optimizers, and observables. The held-out forecast for the 7B model in Figure 4 is potentially useful.

Critical weaknesses
  1. The convex-optimization analysis does not establish the proposed scaling law.

    Theorem 1 provides a convergence upper bound under convexity and boundedness assumptions. The basic O(T−1/2)O(T^{-1/2}) rate is standard. Such an upper bound does not establish the actual loss asymptotics, its tightness, or a common normalized trajectory across model sizes. Moreover, the quantities appearing in the bound depend on model size, and their scaling is not established. The transfer from convex gradient descent to nonconvex neural networks trained with Muon or AdamW also lacks a sufficient theoretical justification. Consequently, the theorem provides limited support for the central explanation of collapse.

  2. The “any-time-any-scale” claims are substantially broader than the demonstrated scope.

    The proposed transformation introduces a burn-in offset, while the experimental recipe requires sufficiently large models or training budgets and fixes several aspects of the training configuration. These restrictions should be reflected in the claims. The paper needs a clearer definition of universality, including the admissible model families, training regimes, observables, and approximation errors.

  3. The assumed separation of training progress and model scale is insufficiently justified.

    The final law represents reducible loss as a compute-dependent amplitude multiplied by a common trajectory Φ(x)\Phi(x). I recognize that this is formulated within a fixed training recipe and that the trajectory can depend on the learning-rate schedule. Nevertheless, the effects of batch size, optimizer, learning rate, regularization, and data distribution are largely implicit in the recipe or fitted coefficients. The paper does not establish when these effects preserve the proposed separation or when scale-dependent changes in trajectory shape become important. Fitting a common Φ(x)\Phi(x) is insufficient to establish its predictive validity across configurations.

  4. The reported predictive accuracy requires more reproducible evidence.

    Figure 4 reports a validation-loss prediction error below 0.5%, but the complete fitting and evaluation pipeline is not sufficiently transparent for independent verification. Please provide the training and fitting code, numerical learning curves, parameter constraints, preprocessing and interpolation procedures, and exact fitting/evaluation splits. In particular, it should be possible to verify that no information from the target model’s trajectory or endpoint enters any stage of fitting or model selection.

  5. The multimodal extension lacks a corresponding analysis of joint training dynamics.

    Section 6 specifies a weighted combination of text and image losses and presents multimodal experiments. However, the theory does not characterize interactions between these objectives, their relative weights, modality mixtures, or shared parameters. Applying a similar fitted functional form to language and multimodal models provides empirical evidence, but does not establish a common optimization mechanism. The claimed unification needs a more precise meaning and stronger justification.

  6. The validation protocol is insufficient for the breadth of the universality claim.

    Although the paper includes multiple experimental configurations, much of the evidence consists of normalized curves fitted within each configuration. The clearest held-out forecasting example is the 7B prediction in Figure 4. A broader claim requires systematic held-out-scale and cross-configuration evaluations, comparisons with existing collapse methods and simpler trajectory models, and uncertainty estimates. Ablations should also isolate the contributions of the proposed hyperparameter scaling, time offset, and closed-form trajectory.

Questions for authors
  1. What additional assumptions make the convex upper bound a tight description of neural-network loss, and what establishes the required relationships between its model-dependent coefficients across scales?

  2. Which components of the framework are shared across configurations, and which must be refitted? Can the authors specify and experimentally test the boundaries of the claimed universality?

  3. Does the proposed closed-form trajectory improve held-out prediction over directly estimating Φ(x)\Phi(x) from reference curves? What are the separate contributions of the time offset and hyperparameter-scaling rules?

  4. How does the framework account for modality mixtures, loss weights, and interactions between text and image objectives?

  5. Will the authors provide code and numerical training trajectories sufficient to reproduce the reported forecasts and compare them against relevant baselines?

Reviewer_9LahOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to combine the standard end-point scaling law with loss-curve collapse to predict full training trajectories. The authors claim to derive a closed form function for the universal trajectory. The authors then provide empirical results across various architectures, optimizers, and hyperparameters.

Critical strengths
  • The goal of extending collapse and scaling laws to benchmark metrics is a very important and relevant problem.
  • The scale of the empirical evaluation is quite substantial.
Critical weaknesses
  1. There is very limited novelty. Qiu et al. (2025) connect collapse to the power-law structure of scaling laws. Bergsma et al. (2026) show collapse under practical recipes and use it for prediction. Theorem 1 is the standard O(1/sqrt(T)) convex rate -- the learning-rate scaling is from Bu et al. (2026) and the weight-decay scaling follows directly from Wang & Aitchison (2025)’s fixed-timescale rule. The proposed closed-form is not compared against existing schedule-aware trajectory laws, Tissue et al. (2026) and Luo et al. (2025) [1].
  2. There are significant evaluation issues. (i) Section 3.1: "Every intermediate-point law shares Beta and L*" -- this appears to be imposed by the joint fit (Eq. 3.3), rather than tested. The authors should fit them independently per x. (ii) T* and L* are tuned to maximize collapse (Eq. 2.3), but the fitted values and the fraction of training excluded are not reported. (iii) There is no quantitative collapse metric. (Qiu et al. 2025 measure collapse against seed noise). (iv) There is only one forecast (Fig. 4) which extrapolates in N and reports no baseline. (v) The 1/sqrt(T) rule is not contrasted with alternatives (e.g., μP transfer, Power Lines [2]).
  3. The paper is not well-written and there are significant discrepancies. The text is very hard to comprehend and the structure of the paper is very hard to follow. The figures are too small (e.g., Figures 7 and 8). There are also discrepancies. For instance, Figure 3 has x axis FLOPs in the range of 10^19-10^21, but Figure 7-8 shows the range to be 10^13-10^16.

References:

[1] Luo et al. (2025). A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules.

[2] Bergsma et al. (2025). Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training.

Questions for authors

Q1. What is RACE? Is it Du et al. (2026)'s LLM-judge framework for scoring long research reports or Lai et al. (2017)'s reading comprehension benchmark?

Reviewer_McKtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified scaling framework that combines endpoint scaling laws with a model of training dynamics. Within fixed token-per-parameter scaling families and suitable training configurations, it aims to predict both performance across model scales and the full training trajectory. The framework extends beyond loss to downstream evaluation metrics and parameter norms, and is examined on dense language models, MoE models, and multimodal models.

Critical strengths

The authors conduct an extensive set of experiments to support the proposed decomposition of scaling behavior and training dynamics. A particular strength is the breadth of the framework: the authors demonstrate related scaling behavior across losses and downstream metrics, dense and MoE architectures, and text-only and multimodal training. This provides valuable evidence for a shared perspective across settings that are often studied separately. The substantial experimental effort is an important contribution of the work.

Critical weaknesses

My main concern is the lack of a systematic evaluation of extrapolation accuracy. Figure 4 demonstrates prediction of a 7B model using runs ranging from 1.18B to 4.16B, and Section 3.2 reports a relative difference below 0.5%. However, broad extrapolation reliability is not sufficiently established by a single held-out target scale. For example, assessing reliable 10× extrapolation in training compute would require evaluation across multiple source–target combinations. Two practical settings are particularly important: predicting larger-scale runs from smaller-scale experiments, and predicting later training behavior from early checkpoints—for example, using the first 5,000 steps to predict performance at 50,000 or 500,000 steps. The paper should distinguish these two settings and clarify whether early-to-late forecasting is supported, while systematically evaluating extrapolation accuracy within its intended scope.

Much of the evidence is presented through fitted or collapsed curves, which do not directly quantify held-out prediction errors—for example, whether these are within 0.5% or 2%. The authors should report concrete error values in tables across multiple target points and extrapolation factors, with clearly defined error measures and comparisons against relevant methods under comparable conditions. This would make it easier to assess both the practical reliability of the method and how its accuracy changes as the extrapolation distance increases.

Closely related to extrapolation is broad applicability, another important aspect of the practical value of scaling laws. Although the authors examine many settings, it remains unclear how well the framework applies to experiments conducted by other researchers. Previous scaling-law studies have released checkpoints or training trajectories that could support such evaluation. Testing the framework on these external runs, where its stated applicability conditions are met, would strengthen the evidence for its applicability beyond the authors’ own experiments. Addressing these two concerns would strengthen the paper and make its contributions more convincing and useful to the community.

Finally, while the breadth across observables, architectures, and modalities is valuable, the term “universal scaling law” may still be somewhat stronger than the current evidence warrants. The requirement to keep D/N fixed within each scaling family, together with the training conditions stated in Appendix A, should be explicitly discussed in the Limitations section to clarify the scope of the framework.

Questions for authors

Please see the concerns raised in Critical Weaknesses.

Regarding reproducibility, do the authors plan to release their code, training trajectories, and model checkpoints or weights? Making these resources available would benefit the broader research community by enabling others to reproduce, apply, and build upon this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.