acceptodds
Under review as a conference paper at ICLR 2027

Unified Neural Scaling Laws

Abstract

We present a functional form (that we refer to as a Unified Neural Scaling Law (UNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number of model parameters, training dataset size, number of training steps, and various hyperparameters) for various architectures and for each of various tasks within a varied set of upstream and downstream tasks. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, neural scaling laws, neural scaling law, broken neural scaling laws, broken neural scaling law

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gpfkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UNSL, a multivariate curve family for neural scaling surfaces that represent regime transition, bottlenecks, irreducible-performance limits, overfitting, and harmful hyperparameter settings. UNSL is evaluated on held-out extrapolation using RMSLE on reused vision and language scaling data.

Critical strengths
  1. Multivariate scaling extrapolation is more useful than fitting one-dimensional parameter count curves. The paper evaluates held-out points rather than relying only on visual fit.

  2. The appendix reports per-task RMSLEs. Authors also use validation scheme to prevent points from being larger than the validation point in all input dimensions.

Critical weaknesses
  1. Most large-scale vision/language evidence is reused from prior studies, while several demonstrations of special behavior use small MLPs, one-epoch training. UNSL do not establish reliability for modern training runs, data mixtures, and architecture families.

  2. Reporting the percentage of tasks "won" hides effect size and instability. Since hyperbreak counts and related choices are selected by validation and fitting uses many optimization steps, these omissions are consequential.

  3. Limited set of held-out points from the same pre-existing scaling surfaces used for fitting are evaluated. This does not systematically test multiple extrapolation distances. Can you report results under a standardized extrapolation stress test?

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer gpfk

We thank you for the positive and constructive review.

Weakness 1 (reused data; small MLPs). The reused data are modern, large-scale runs across architecture families, and small MLPs are used only in controlled studies. We reused published scaling data on purpose, because those runs could not have been generated or tuned for UNSL, and every form is compared on the same data. The runs cover Transformers, linear RNNs, ViT, MLP-Mixer and ResNets, and they include compute-optimal and constant learning-rate schedules as well as multi-epoch training (scales are in our "General response to all reviewers" comment). The controlled studies each need many full training runs, which would be far more costly at LLM scale. The non-monotone effect of overfitting is also evaluated at scale, in multi-epoch LMs up to 8.7B (Fig. 27) and in JFT vision models (Fig. 17). The benchmark does not vary data-mixture proportions, so the paper makes no claim about them.

Weakness 2 (effect size; instability). The effects are large and consistent, and model selection never sees the held-out points, so it cannot favor UNSL. Effect sizes complement "% of tasks won", and the revision reports them in Sec. 4 (Table 3, also in the general response). The baselines' extrapolation errors are 3.4x (CF), 28x (DC), 2.0x (A1), 1.44x (A2) and 1.18x (A3) UNSL's, where A1-A3 are the paper's ablation functional forms, which are simpler versions of UNSL. All paired p≤0.01p \le 0.01.

On instability, we make four points.

  1. The number of hyperbreaks nn of each of A1, A2, A3 and UNSL, which controls its number of parameters, and the "related choices" you mention are selected only on validation points taken from the fitting points (triangles in the paper's plots). These validation points are disjoint from the held-out points used to evaluate extrapolation (circles), and the held-out points are never used for selection. No fitting point exceeds a validation point in every input. The protocol is given in App. D and does not differ between A1, A2, A3 and UNSL.
  2. Fitting runs several random initializations to convergence (App. D), which takes many optimizer steps, and keeps the one with the lowest loss on the fitting points. The held-out points are never used to choose among the initializations. Choosing the best of several initializations is standard for scaling-law fits (e.g. Hoffmann et al., 2022).
  3. A1, A2, A3 and UNSL are all fit and validated with the same procedure, so the selection treats them alike and does not favor UNSL.
  4. The fit is also stable when the fitting data change. Across a roughly 100x range in the number of fitting points, from 931 down to 9, UNSL's extrapolations stay accurate (App. G, Fig. 5).

Weakness 3 (extrapolation distance). The held-out points span a range of distances, and a controlled study stress-tests extrapolation in several inputs at once. In the benchmark, held-out points sit at the large-scale end of each surface, which matches how scaling laws are used in practice, namely to forecast larger runs of the same setup. Reading off the paper's plots, the 8.7B model is held out entirely, 2x beyond the largest fitted model (Fig. 27, data-constrained LM runs). The largest Chinchilla models are held out, and in the upstream panel the ~16B model is about 2.4x beyond the largest fitted model (Fig. 32, Chinchilla and constant-learning-rate LM runs). The ~7B Transformer and HGRN2 models are fit only on their early training steps and extrapolated to about 6-20x their last fitted step, depending on the model and task (Fig. 32). In the controlled study, UNSL stays accurate an order of magnitude beyond the fitting range (App. H, Fig. 6).

The reused runs are large-scale and could not have been tuned for UNSL, the effect sizes are large (Table 3), the fits are stable, and the held-out data include models at least 2x larger than any fitted one and points about an order of magnitude beyond the fitting range (Figs. 27, 32 and 6). We believe this addresses each weakness the review lists and strengthens the case for acceptance.

Reviewer_RMEtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new, highly parameterized functional form for neural scaling laws. The form is shown to obey several reasonable desiderata that have been observed in prior literature. Empirically, the functional form is shown to yield better extrapolation performance than other scaling laws.

Critical strengths
  1. The primary strength of the proposed scaling law is its strong empirical performance. It dramatically outperforms the CF and DC baselines, and even outperforms its own ablations. This indicates that the proposal holds strong practical relevance.
Critical weaknesses
  1. The primary weakness in my view is the lack of comparable baseline scaling laws to compare with. UNSL has many parameters unlike CF and DC, which are simpler. A fairer baseline to compare with would be some with a similar number of parameters. One possibility is even fitting a small neural network to the scaling law and comparing its extrapolative performance to UNSL. The ablations are not a sufficient comparison.
  2. This is a less important concern, but there is no formal theorem deriving UNSL as a unique functional form from the desiderata. This makes it hard to assess whether UNSL is unique or minimal; there could be many scaling laws obeying the deiserata, and UNSL could be an arbitrary member of this set.
Questions for authors
  1. How does UNSL compare on extrapolation performance to a scaling law with a similar number of parameters?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RMEt

Thank you for your review and for noting UNSL's strong empirical performance. You ask how UNSL compares with forms that have a similar number of parameters, such as a small neural network. The submission already contains results that answer this. We summarize them below, and the revision states them in Sec. 4.

Weakness 1 and Question 1. The functional forms A1, A2, A3 and UNSL are each fit with their best number of parameters, and UNSL still extrapolates better, including against the small network A1.

  1. The number of parameters of A1, A2, A3 and UNSL is tuned. In each of these four forms, the number of hyperbreaks nn controls it. For each of the four forms, nn is selected on validation points taken from the fitting points (triangles in the paper's plots), and these are disjoint from the held-out points used to evaluate extrapolation (circles). The held-out points are never used for any selection, and the protocol, given in App. D, is identical for A1, A2, A3 and UNSL. So each of the four is compared at its own best number of parameters. This is a stronger control than forcing equal counts, because each baseline gets the size that works best for it. As an additional check, in Table 8, where both have the same number of parameters, UNSL also extrapolates better than its ablated version.
  2. A small neural network is among the baselines. In log-log space, A1 is a one-hidden-layer softplus network with a linear skip connection. Each of A1's hidden units is a hyperbreak, so A1's width, and hence its number of parameters, is chosen on validation in the same way as the number of hyperbreaks of A2, A3 and UNSL. UNSL extrapolates better on 28 of 32 tasks, and A1's error is 2.0x UNSL's (p = 2e-6, see our "General response to all reviewers" comment). The ±\pm intervals that the paper's tables report with each error separate in UNSL's favor on 23 tasks and in A1's favor on 3.
  3. The gains come from structure. Each form is fit with its validated number of parameters, so size does not explain the gains. Error still falls at every step that enforces more desiderata. Over the 24 tasks that have all four forms, the geometric-mean error is 2.28% for A1, 1.78% for A2 (which adds floor and bottleneck terms), 1.55% for A3 (which adds ceilings and hyperparameter terms) and 1.32% for UNSL (which adds the overfitting term).

We include CF and DC because they are the forms used in practice. Their large errors come from missing structure. Neither has ceilings such as random-guess error, and both are monotone in every input, so they cannot fit the overfitting upturns in the data, for example in the JFT vision fits of Fig. 17.

Weakness 2 (uniqueness). We do not claim that UNSL is unique, and the revision says so explicitly. UNSL is also not arbitrary. The desiderata are empirical properties of scaling curves, covering limits, bottlenecks, transitions and how opposing effects interact, and UNSL is built to satisfy them. They constrain the form but do not single it out. As with previously proposed forms, such as the Kaplan, Chinchilla and data-constrained scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023), the support for UNSL is empirical. Each part of UNSL enforces a particular desideratum (Table 1, with App. J explaining how). Each group of construction steps in Sec. 2.2 (Step 2, Steps 3-4 and Step 5) improves extrapolation, since removing them one group at a time, from UNSL to A3, A2 and A1, raises average extrapolation error at every stage (point 3 above).

Each of A1, A2, A3 and UNSL is compared at its own best number of parameters, and UNSL beats A1, the small neural network the review suggests, on 28 of 32 tasks. In our view, this addresses the primary weakness the review names and warrants revisiting the rating.

Thank you for the clarification. I’m now more convinced that baseline comparisons were appropriately conducted, and have raised by rating accordingly.

My concern on uniqueness still remains though. Unlike previously proposed scaling laws, which have a simple and interpretable form, UNSL is quite complicated. This, it requires some notion of minimality and uniqueness. Since UNSL does not uniquely satisfy the desiderata, at least it would be helpful to characterize the space of scaling laws that do satisfy the desiderata and argue that UNSL is the least complicated choice.

Two laws can satisfy all eight desiderata and still give different predictions between regimes. Near the best attainable error, both must reduce to a floor plus one power-law term for each limiting resource, such as model size or training data. The desiderata also constrain how error approaches a ceiling, how a bend shifts when another input changes, and how competing effects can reverse a trend.

Once the power laws on either side of a bend are fixed, Eq. (3) uses one parameter, f, to set how gradually one gives way to the other. In Figure 7, fitting f gives a better extrapolation than fixing it at 1. We could save a parameter by using the same f at two bends. That would assume the value fitted at one bend also applies at the other, a relationship the desiderata leave open.

Test error can turn upward with further training as the model overfits, or when the learning rate or the standard deviation of the initial weights gets too large. In each case we add a rising term to a falling one. Their sum determines where the minimum occurs and how rounded it is. To make a positive trend u level off at a ceiling v, we use uv/(u+v). We use these same two rules throughout UNSL to combine the component trends and their limits without fitting extra weights where they meet

Reviewer_BFvSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The work proposes a single template for all scaling law forms, based on a multivariate extension of Broken Neural Scaling Laws. Like other scaling laws, UNSL takes in m dependent variables or inputs (e.g. number of parameters) and predicts performance (e.g. loss L). The specific proposed form is based on a multivariate extension of broken scaling laws, and should flexibly extend to arbitrary inputs. The authors spend most of the paper explaining the desiderata based on which the form is designed, proposing alternate forms for ablations, and considering the practical use of this form, measuring its accuracy under extrapolation (with results relegated to the appendix).

Critical strengths

S1. The work seeks to address the current state of scaling laws, in which researchers custom-design new scaling laws when needed, to incorporate axes of scaling introduced by novel architectures or training algorithms. This is indeed a phenomenon of scaling law research, and a verifiable universal scaling law form would be helpful in being able to draw connections between our understanding of varied algorithms and architectures.

S2. There is broad experimentation to investigate the usage of the proposed scaling law form in the Appendix. (Caveats to this strength in W3)

Critical weaknesses

W1. As stated by the authors, incorporating more dependent factors and variables increases the ability of a scaling law to represent the true underlying causal mechanism, since omitting any causal factor inevitably results in noise. However, this statement does not take into account the actuality of scaling law fitting procedures in empirical science, where scaling laws are fit to a very limited set of observations. Each independent datapoint may be prohibitively costly, so statistical power is capped by the computational budget. Defining a form that seeks to be unified across settings may indeed increase the ability of the law form to fit the true underlying pattern, but it also substantially increases the likelihood of overfitting, as new parameters add degrees of freedom which would require exponentially more datapoints to achieve confidence for. It seems that the construction of UNSL may be designed to address this issue, but it's very difficult to tell because of the extremely non-linear writing (W2)

W2. The paper is not readable. It is extremely difficult to follow, not just because there is a high density of mathematical content and a significant demand for prerequisite knowledge. The paper is actually badly written: the structure of the explanation is disorganized with an excessive number of forward references, nearly every sentence in many sections of the paper has a semicolon joining two clauses, and many newer concepts or specific aspects of UNSL are under-explained. This feels like a paper which was cut down for submission but was not proofread by any non-author reader. Even though I feel that the paper has genuinely important contributions, I also feel that it is so significantly below the writing quality bar right now, it cannot be accepted without major rewrites. I would not have considered this paper anything other than an early draft, not suitable for dissemination.

W3. All results demonstrating practical usefulness and extrapolation are relegated to the Appendix, which is not required reading. Though these results are interesting and extend through several settings, they are almost entirely at very small scale. While I don't believe that scaling law work necessitates extremely high computational budgets, the majority of claims are made with models around 10M parameter or smaller scale, which is a very low threshold, as even academic research on scaling laws typically considers models in the 1B-7B range for its main claims.

Questions for authors

The writing quality alone makes me very confident in my appraisal, and I don't believe that there are any questions with answers that would really change my scores. But in addition, the information which is included in the main text is insufficient for a conference-level contribution, with nearly all evidence of soundness and applicability left to the extremely long appendix.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BFvS

Thank you for the detailed review. We share the view in S1 that a verifiable unified form would be useful, and we appreciate your remark on the breadth of the experiments (S2).

W2 (writing). Thank you for the specific points on the writing. The revision addresses each of them without changing the method or the results. Sec. 2 now opens with a short roadmap and a box that defines the general terms (performance and misperformance limits, bottlenecks and oppositional forces) before first use. It then builds UNSL in five steps, each using only earlier ones, and defines the other new concepts where they are introduced. Apart from Table 1 and one sentence at the start of Sec. 2.2 that points to supporting material, every forward reference and appendix pointer in Sec. 2 is removed. Every semicolon-joined sentence in the main text is split, so none remain.

W3 (scale). The benchmark results, which carry the paper's main claim, do not rest on ~10M-parameter models. All 32 tasks behind Table 2 are large-scale runs from prior work, which cover and go beyond the 1B-7B range you mention. They include LMs up to 16B and JFT-pretrained vision models trained on up to 3B images, as detailed in our "General response to all reviewers" comment. Small MLPs appear only in controlled studies (e.g. Figs. 4, 6, 9 and 11), each of which needs many full training runs and would be far more costly at billion-parameter scale. The per-task tables (Tables 4-7) were in the appendix only because of the 9-page limit, and the revision uses the 10th page to move them into Sec. 4 together with an effect-size table (see the general response).

W1 (overfitting with limited data). UNSL is built to avoid this. The conditional-entropy argument in Sec. 1 concerns the unlimited-data limit, and the revised introduction says so explicitly. With finite data, UNSL controls overfitting in four ways.

  1. The desiderata only add constraints. They fix the limits and asymptotes of the form, which is where extrapolation happens.
  2. The evaluation penalizes overfitting, because all errors are measured on held-out points beyond the fitting range. There, UNSL beats A1 on 28 of 32 tasks, A1's error is 2.0x UNSL's, and the error falls with each added constraint. On the 24 tasks with all four forms, A1 has the highest error, then A2, then A3, and UNSL has the lowest.
  3. Size is selected on validation points. For each of A1, A2, A3 and UNSL, the number of hyperbreaks nn, which controls its number of parameters, is chosen on validation points taken from the fitting points. These are disjoint from the held-out points used to evaluate extrapolation, which are never used for this choice. No fitting point exceeds a validation point in every input. App. D describes this protocol, which is the same for all four forms.
  4. UNSL is data efficient. It stays accurate with as few as 9 fitting points (App. G, Fig. 5), and it extrapolates an order of magnitude beyond the fitting range in multiple inputs at once (App. H, Fig. 6).

In short, the uploaded revision rewrites Sec. 2 and moves the per-task results into Sec. 4, the benchmark includes LMs of up to 16B parameters, and on held-out points, where any overfitting would show up, UNSL has the lowest average error of all the forms we compare. We believe this resolves the concerns behind the current rating and merits a reassessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers, who all consider the goal important. RMEt also notes that UNSL "dramatically outperforms the CF and DC baselines".

  • None of the benchmark's claims rest on ~10M-parameter models. They rest on runs of up to 16B parameters, including the 1B-7B range BFvS mentions.
  • The number of fitted parameters of each functional form A1, A2, A3 and UNSL is tuned on validation points disjoint from the held-out extrapolation points. A1 is a small neural network, and UNSL still beats it on 28 of 32 tasks.
  • The Chinchilla law (CF) has 3.4x UNSL's extrapolation error, and the data-constrained law (DC) has 28x.
  • The revision (now uploaded) addresses the presentation concerns, and our reply below lists the changes.

All numbers come from tables and figures already in the submitted paper, so each can be checked there. Errors are root-mean-square log errors (RMSLE), quoted in % as rough relative errors. Table, figure and appendix numbers refer to the revised PDF.

What the paper contributes. UNSL is one functional form for any set of inputs (model size, data, training steps, hyperparameters). Unlike CF and DC, it captures non-monotone effects such as overfitting and too-large learning rates or initializations. It rests on the additive symmetry of Sec. 2.1, under which non-monotone transitions behave like a sum of two power laws, bending exactly once with fixed sharpness. It also shows when extrapolation fails, since a transition is predictable only if it lies near the fitted data (Sec. 5).

1. Scale. All 32 benchmark tasks are large-scale runs from prior work. They include language models from 7M to 16B parameters (Chinchilla runs, multi-epoch data-constrained runs, and Transformer and HGRN2 LMs up to 7B) and JFT-pretrained ViT, MLP-Mixer and BiT models trained on up to 3B images.

For example, Fig. 27 (Table 7) shows one UNSL fit across 20 model sizes (7M-8.7B), with multi-epoch overfitting upturns and a fully held-out 8.7B model. UNSL's extrapolation error on this task is 0.78%, versus 6.24% for DC. Small MLPs are used only in controlled studies (e.g. Figs. 4, 6, 9 and 11) that each need many full training runs.

2. Effect sizes (now Table 3). UNSL extrapolates better than every baseline. The ratio is the geometric mean of baseline error over UNSL error, and p is from a one-sided paired Wilcoxon signed-rank test on log errors.

UNSL vs.UNSL betterratiop
CF (Chinchilla form)8/83.4x0.004
DC (data-constrained form)24/2428x6e-8
A1 (one-hidden-layer NN in log-log space)28/322.0x2e-6
A2 (A1 + floor and bottleneck terms)26/321.44x4e-5
A3 (A2 + ceilings and hyperparameter terms)17/241.18x0.01

A1, A2, A3 and UNSL are functional forms built from KK, a multivariate broken power law (Eq. 1). A1-A3 are ablations of UNSL that each add the listed terms, and UNSL adds an overfitting term. The floor a0a_0 is the irreducible error, and bottleneck terms give the limit when one input, such as data, is the bottleneck. Ceilings cap the error, e.g. at random guessing, and hyperparameter terms model too-large learning rates or initializations.

A task is one scaling surface (one model family on one metric). There are 23 vision and 9 language tasks. The 8 single-epoch language tasks repeat no data, so UNSL's overfitting term vanishes and UNSL coincides with A3. CF applies there. The other 24 repeat data, so DC (which reduces to CF without repetition) and A3 apply.

On the 24 tasks with all four forms, error falls at every step that enforces more desiderata (eight empirical properties of scaling curves), from 2.28% (A1) to 1.78% (A2), 1.55% (A3) and 1.32% (UNSL).

3. Number of parameters (RMEt). A1, A2, A3 and UNSL each get their best number of parameters. In each form, the number of fitted parameters is set by the number of hyperbreaks nn (smooth bends in the scaling surface, i.e. hidden units in A1). We tune nn on validation points taken from the fitting points (triangles in the paper's plots), disjoint from the held-out points (circles). As an additional check, Table 8 shows that at an equal parameter count UNSL also extrapolates better than its ablated version.

AuthorsOpenReview ↗

Revision uploaded: summary of changes and new numbering

We have uploaded the revision. Sec. 2 now opens with a short roadmap and a box that defines the general terms before first use. Apart from Table 1 and one sentence at the start of Sec. 2.2 that points to supporting material, Sec. 2 has no forward references or appendix pointers. Every semicolon-joined sentence in the main text is split, so none remain. The per-task results (Tables 4-7) and the effect-size table are now in Sec. 4, using the 10th page. Sec. 4 also states the data scale, that the number of hyperbreaks (and hence of parameters) of each of A1, A2, A3 and UNSL is tuned on validation points disjoint from the held-out points, and the A1 and equal-parameter comparisons. The introduction's entropy argument is now qualified for finite data, and a non-uniqueness remark is added to Sec. 2.

The appendices are now lettered (submitted App. 7 -> A, ..., App. 17 -> K, App. 19 -> L, App. 20 -> M), and the tables of App. 18 are now in Sec. 4. Tables are renumbered (submitted 3 -> 8, 4 -> 5, 5 -> 4, 6 -> 7, 7 -> 6), and Table 3 is the new effect-size table. Our responses use the revised numbering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.