acceptodds
Under review as a conference paper at ICLR 2027

Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

Abstract

Neural scaling laws are foundational for language model development, yet the standard formulation systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. We show that this interaction is directly measurable in the loss surface, and capture it with the Skaling law (/ska.liN/}), a coupled functional form that extends Chinchilla with a single interaction exponent between model capacity and data. On two large training grids, this one extra parameter reduces the Mean Absolute Percentage Error (MAPE) by roughly – on interpolation and extrapolation, and by up to an order of magnitude when fitted on sparse grids. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law matches or exceeds the accuracy of Chinchilla fitted on the full grid while using up to less compute. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_us36OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the interaction between model size and training data in neural scaling laws. It introduces an outer exponent into the additive Chinchilla formulation to capture this interaction while retaining an analytical compute-optimal allocation. The authors evaluate predictive accuracy across several extrapolation settings and propose an L-shaped sampling design to reduce the cost of fitting the scaling law.

Critical strengths
  1. The paper offers a clear and interpretable perspective. It examines how model size and data interact to explain systematic prediction errors in additive scaling laws, establishing a direct connection between the problem and the proposed formulation.
  2. It captures coupling through a single additional parameter while preserving an analytical compute-allocation rule, offering a useful balance between expressiveness and simplicity, which makes the main predictive evaluation systematic.
Critical weaknesses
  1. The novelty of the functional form is limited. Section 5 acknowledges that Busbridge et al. already used the same formulation. The contribution therefore lies primarily in the analysis and experimental design, and the paper should more clearly establish what new understanding it provides beyond the existing formulation.
  2. The advantage over Chinchilla depends partly on the fitting procedure. In Table 9, dominated-pair fitting reduces Chinchilla’s far-extrapolation error on the full Farseer grid from 2.46% to 0.79%, outperforming both standard Skaling (2.32%) and Skaling with the same fitting modification (1.55%). This suggests that the main comparison does not fully separate the effects of the functional form from those of the fitting procedure.
  3. Better loss prediction does not necessarily imply better compute allocation. The far-extrapolation evaluations contain only seven and three additional runs, respectively, and the recommended configurations at substantially larger compute budgets are not directly validated. Table 5 also shows smaller and less consistent gains on the original Chinchilla data, leaving the generality of the conclusions uncertain.
Questions for authors

Can the authors directly compare the configurations recommended by different laws at a held-out compute budget, rather than evaluating loss prediction alone?

3 comments

Thank you for the careful review. Your three concerns are fair. Below we (1) state what the paper adds beyond Busbridge et al., (2) show that the advantage over Chinchilla holds when Chinchilla uses its best fitting procedure, and (3) check the allocation directly, including at a held-out compute budget.

W1. Novelty of the functional form. We agree that the form is not new. We will credit Busbridge et al. (2025) in the introduction and in Sec. 3.1, not only in Sec. 5. They use this form for the supervised losses inside their distillation scaling law and note that it fits better than tied forms. They do not compare it with other laws under a controlled protocol, nor study the N–D interaction. What our paper adds:

  1. Evidence that the interaction exists, independent of any law (Sec. 2). The mixed derivative of the loss surface, estimated without a parametric model, is non-zero and mostly negative across the Farseer grid. Every additive law forces it to zero, while Skaling with 0<k<10 < k < 1 reproduces its sign (App. A.2).
  2. A controlled comparison with Chinchilla and Farseer, using the same objective, on four datasets and up to four extrapolation regimes per dataset, plus iso-ratio compute extrapolation (Tables 1, 3, 5).
  3. When the coupling matters. The fitted kk indicates it: gains are large where k≈0.3k \approx 0.3–0.450.45 (Farseer, SK-Grid) and smaller where k≈0.77k \approx 0.77–0.900.90 (Table 5).
  4. Sparse grids. An L-shape of cheap runs matches full-grid Chinchilla at 5–10× less compute. This works only for a form that represents the interaction (Sec. 4.2).
  5. Allocation. On Farseer, Skaling's compute-optimal ratio follows a model-free estimate of the frontier, with the same sign of trend; Chinchilla's has the opposite sign (App. A.3). This holds when the laws are fitted only on cheaper runs (see Q).

W2. Functional form vs. fitting procedure. In the main comparison, Chinchilla and Skaling share the same optimizer, objective and fitting budget, and both fits converge, so the difference comes from the form. App. F tries an alternative objective for both laws. Even with whichever of its two fits is better, Chinchilla beats default Skaling in only 1 of 16 cells (two main grids, full and L-shape, four regimes): the one you point out, far extrapolation on full Farseer (0.79 vs 2.32%). The pairwise objective mostly reweights the grid: runs in its corners (large models trained on few tokens, small models trained on many tokens) belong to few dominated pairs, so they count less. On Farseer, the runs with D/N<2D/N < 2 or D/N>300D/N > 300 are 20% of the data but get 12% of the weight. These corners are where Chinchilla errs most (Fig. 1), so it gains on the far runs but loses on larger DD (1.98 → 2.61%), and on the L-shape its far error stays 5–11× above Skaling's. See also our reply to Reviewer 4ngE (W2).

W3. Loss prediction vs. compute allocation. Both laws derive the allocation from their fitted loss surface with the same formula (App. B), so a law that predicts the loss better, inside and beyond the grid, should also place the optimum better. We also check the allocation directly, not only through loss prediction. We do not claim validated recommendations at 102510^{25} FLOPs; Fig. 5 only shows how far the two laws diverge. Three points support the allocation:

  • Chinchilla's allocation is already wrong inside the grid (App. A.3, Fig. 5). On Farseer, we estimate the compute-optimal frontier from the runs alone, using GP and MLS surrogates with no parametric law. The optimal tokens per parameter decrease with compute (D∗/N∗∝C−0.14D^*/N^* \propto C^{-0.14} and C−0.15C^{-0.15}). Fitted on the full grid, Skaling follows this trend (−0.11-0.11), while Chinchilla predicts the opposite sign (+0.03+0.03). A law that is wrong on the runs it was fitted on has no reason to be right beyond them. The held-out check in Q confirms this.
  • The far sets are small, but they are not the only test. Table 3 evaluates 112 held-out high-compute Farseer runs, where Skaling's error is 3.9× lower than Chinchilla's. The Farseer Ext. N and Ext. D sets hold another 36 and 66 runs.

On Table 5: the gains shrink where the fitted coupling is weak (k≈0.77k \approx 0.77–0.900.90), as the nested form predicts. On the Chinchilla data, the two laws are within 0.12 MAPE points of each other. Skaling thus costs little when the coupling is weak and helps when it is strong, and the fitted kk says which case applies.

Q. Comparing recommended configurations at a held-out budget. Yes. We fit both laws only on the Farseer runs below a compute budget C0C_0. We then compare their recommendations at 3×10213 \times 10^{21} FLOPs, the largest budget of the grid, which they never see, with the model-free frontier of App. A.3. Here mm is the exponent in D∗/N∗∝CmD^*/N^* \propto C^m:

Laws fitted on runs withmm, Skalingmm, ChinchillaD∗/N∗D^*/N^* at 3×10213 \times 10^{21}, SkalingD∗/N∗D^*/N^* at 3×10213 \times 10^{21}, Chinchilla
C<1020C < 10^{20} (197 runs)+0.08+0.352641075
C<3×1020C < 3 \times 10^{20} (271 runs)−0.02+0.26158588
C<5×1020C < 5 \times 10^{20} (314 runs)−0.07+0.20129439
C<1021C < 10^{21} (352 runs)−0.10+0.14114350
Model-free frontier, all 404 runs (GP / MLS)−0.14 / −0.1585 / 93
  • Skaling approaches the frontier as C0C_0 grows. Its slope turns negative from C0=3×1020C_0 = 3 \times 10^{20}, and at C0=1021C_0 = 10^{21} it recommends 114 tokens per parameter, against 85–93 for the frontier.
  • Chinchilla's slope has the wrong sign at every C0C_0. It recommends more tokens per parameter as compute grows, the opposite of the data, and overshoots the frontier at 3×10213 \times 10^{21} by 4–13×.

The extrapolation is short: the grid ends at 3.5×10213.5 \times 10^{21} FLOPs, so 3×10213 \times 10^{21} is only half a decade above the largest C0C_0. This does not show that Skaling's allocation still holds at much larger compute, which we do not have the budget to test. But it is direct, held-out evidence that Skaling allocates compute better than Chinchilla. We will add this table to App. A.3.

Dear Reviewer us36,

As the discussion closes today, we would value your feedback on our reply. In short, we addressed your three concerns and your question:

  • Novelty (W1). We agree that the form is not new. We will credit Busbridge et al. (2025) in the introduction and in Sec. 3.1, and we listed what the paper adds beyond the form: model-free evidence of the N–D interaction, a controlled comparison on four datasets, the fitted kk as a guide to when the coupling matters, the L-shape design, and the compute allocation.
  • Form vs. fitting procedure (W2). Even with its better App. F fit, Chinchilla beats default Skaling in only 1 of 16 cells. The alternative objective mainly reweights the grid.
  • Allocation (W3 and Q). We ran the held-out test you asked for. Fitted only on runs below a compute budget, Skaling's recommended tokens per parameter at 3×10213 \times 10^{21} FLOPs move toward the model-free frontier as the budget grows (114 vs 85–93). Chinchilla's trend has the wrong sign at every budget and overshoots the frontier by 4–13×. We will add this table to App. A.3.

We would be glad to clarify anything that remains unclear. Thank you again for your review.

Reviewer_qWJdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the N–D interaction in scaling laws using a six-parameter formula with good fitting and prediction results. It also provides additional experiments covering 125 training configurations.

Critical strengths

The paper captures the N–D interaction with fewer parameters than Farseer—six instead of nine—and reports comparable or potentially better results. The simpler formulation and additional training experiments are its main strengths.

Critical weaknesses

1. The Farseer baseline needs further checking.

1A. In the Full-grid results of Table 1, Farseer performs substantially worse than Chinchilla on interpolation and N/D extrapolation. This differs from the advantage reported in the original Farseer paper and needs further explanation.

We ran an additional comparison on the public data: excluding N ≤ 200M, holding out the three largest model sizes, and retaining all D values for the remaining sizes, giving 305 training records. We fitted all three formulas directly, separately using English or aggregate BPC, with matching training and evaluation targets. This split differs from Table 1, so this is an additional comparison rather than an exact reproduction.

Errors below are percentages, listed as Train / N extrapolation / Far7 / 25B.

English BPC:

Farseer: 0.128 / 0.153 / 0.343 / 0.416

Skaling: 0.167 / 0.242 / 0.409 / 0.357

Chinchilla: 0.555 / 1.020 / 2.419 / 2.295

Aggregate BPC:

Farseer: 0.485 / 0.297 / 1.394 / 1.698

Skaling: 0.385 / 0.415 / 1.350 / 2.499

Chinchilla: 0.842 / 1.521 / 2.238 / 2.546

Farseer outperforms Chinchilla on every listed metric here. The different split prevents a direct conclusion that the authors’ results are wrong, but the weak Farseer baseline in Table 1 warrants further checking.

1B. The original Farseer evaluation focuses on English validation BPC. These checkpoints were trained predominantly on English, with relatively little Chinese data, so I consider English BPC a more appropriate target for this dataset. The paper may use aggregate BPC, but this is not clearly stated.

For Table 3, I understand that these labels are only reference points. Still, if aggregate BPC is used, total D/N may not reflect the training regime of the Chinese component, which receives much less training data. English BPC would make the comparison clearer and more consistent with the original Farseer evaluation.

2. The reproducibility and transparency of the work need to be strengthened.

The paper does not clearly state whether the trained checkpoints, fitting code, and supporting data will be made available. Providing these materials and the necessary experimental details would help others independently verify the results.

I like the simpler formulation and the additional experiments. If the authors can address the concerns about the correctness of the Farseer reproduction and strengthen the reproducibility and transparency of the overall work, I think this would be a good paper.

Questions for authors
  1. Could you provide the exact Farseer data points(The 302 fitting configurations total ∼5.0 × 10²² FLOPs) used for fitting in Table 1?
  2. Do Tables 1 and 3 use aggregate or English validation BPC?
5 comments

Thank you for checking our Farseer baseline on the public data. You were right: our Farseer fits had not converged. We have redone them. Below are the corrected values, how they compare, and how the results can be reproduced.

W1A. The Farseer baseline. We changed two things in Farseer's fit. The data, folds, objective and search (one basin-hopping run, the same for every law) are unchanged:

  • Normalized N. We divide NN by its median, an exact reparameterization. With raw NN (10810^8 to 101010^{10}), Farseer's coefficients and exponents span many orders of magnitude, and the optimizer stalls.
  • Farseer's own procedure as starting point. We initialize from a simplified version of the stage-wise fit of Li et al. (2025, Sec. 3.2): for each model size with at least 3 training runs, a power law in DD fitted to the loss (not to loss differences), whose exponent and coefficient are then fitted as functions of NN. All nine parameters are then fitted jointly.

Corrected Farseer values for Table 1 (MAPE %):

Farseer, new fitInterp.Ext. NExt. DFar
Farseer grid, full0.46 ± 0.040.33 ± 0.051.28 ± 0.062.63 ± 0.45
Farseer grid, L-shape1.15 ± 0.171.53 ± 0.611.47 ± 0.191.86 ± 0.56
SK-Grid, full0.32 ± 0.090.49 ± 0.350.34 ± 0.222.44 ± 1.77
SK-Grid, L-shape1.13 ± 0.485.24 ± 1.342.30 ± 1.0714.10 ± 5.31
  • Against Chinchilla: Farseer now wins in nearly every cell of Table 1, as in Li et al. (2025) and in your split.
  • Against Skaling, full grids: the two laws are close, and each is better in some regimes.
  • Against Skaling, L-shapes: Skaling is the most accurate law in every regime on both main grids.

Farseer's end-to-end fit is not unique. In every fold, the four runs of Farseer's fit ended in four different minima (180 of 180 runs). Near-optimal fits also extrapolate differently. On a single fold, fits within 1% of the lowest training objective have far errors from 1.9% to 2.4% (Farseer grid, aggregate, full) and from 2.0% to 3.5% (Farseer grid, aggregate, L-shape); fits within 5% range from 8.6% to 16.8% (SK-Grid, L-shape). Li et al. (2025, App. H) report a related difficulty: direct fitting "struggled to converge to an optimal global solution, even with the extensive initialization strategy" of over 30,000 optimization runs. Their stage-wise fit (App. C) instead regresses, for each model size, loss differences L(N,D)−L(N,λD)L(N, D) - L(N, \lambda D) at a fixed ratio λ\lambda. Our CV folds leave holes in each size's DD ladder (12–19% of consecutive steps skip at least one rung), and in the L-shapes several sizes keep fewer than two such differences. We therefore use a simplified version, fitting each size's loss directly, only to initialize the end-to-end fit.

W1A (continued). Overall comparison. To summarize all grids at once, the table below gives the probability that Skaling has a lower error than each law on the same CV fold. It is averaged over four grids, each counted once with 95% bootstrap intervals (Agarwal et al., 2021). 1.0 means Skaling always has the lower error, 0.0 that it never does, and 0.5 that the two laws are even.

P(Skaling has the lower error)Interp.Ext. NExt. DFar
vs Chinchilla, full grid0.95 [0.85, 1.00]0.75 [0.75, 0.75]1.00 [1.00, 1.00]0.80 [0.75, 0.90]
vs Chinchilla, L-shape0.80 [0.67, 0.93]0.80 [0.67, 0.93]0.80 [0.67, 0.93]1.00 [1.00, 1.00]
vs Farseer, full grid0.45 [0.29, 0.62]0.39 [0.24, 0.54]0.39 [0.28, 0.51]0.64 [0.54, 0.72]
vs Farseer, L-shape0.63 [0.50, 0.77]0.55 [0.50, 0.63]0.53 [0.44, 0.63]0.66 [0.56, 0.75]
  • Against Chinchilla, Skaling is better in every setting (0.75–1.00).
  • Against Farseer, full grids, the two laws are very similar Farseer seems to be a bit better on ext-N and D but a bit worse on Far.
  • Against Farseer, L-shapes, Skaling is a bit better.

We will upload the revised PDF with all corrected numbers before the end of the discussion period. It will correct Tables 1, 3 and 5, add this comparison, describe the new fit and Farseer's multiple minima in App. C, and rewrite the paragraph of Sec. 4.2 that starts "More parameters do not mean better extrapolation", whose comparison relied on the unconverged fits. The revised conclusion has three parts:

  • Both laws that model the N–D interaction beat the additive law.
  • Skaling and Farseer are close: even on complete grids; on the L-shapes, Skaling is slightly ahead overall, though not on every grid.
  • Skaling reaches this accuracy with six parameters instead of nine, and keeps a closed-form compute-optimal allocation.

W1B and Q2. Evaluation target. Tables 1 and 3 use the aggregate raw validation BPC (IntelliValSet_Raw). The released columns are not documented, and the main text of Li et al. (2025) does not name the language of its validation set. Without this information, we chose the most general target: the aggregate over both languages. Li et al. report English BPC (their App. D.1), which is the unlabeled column L in the released data. English BPC is also a reasonable target; we will state ours explicitly in Sec. 4.1 and add all results with English BPC to the appendix, next to the other datasets. Every law is fitted and evaluated on the same target, so the comparison is like-for-like. Your English-BPC numbers also give the same ranking against Chinchilla.

On Table 3: like Table 1, it uses aggregate BPC. We will update it with the corrected Farseer fit; the ranking does not change.

W2 and Q1. Reproducibility. Apart from SK-Grid, every result in the paper uses public data. To make the main ones reproducible:

  • The 302 fitting runs (Q1) are the 404 released Farseer runs minus two held-out sets: the 3 largest model sizes (4.5B–6.4B, 36 runs) and the 3 largest data budgets of each remaining size (66 runs). The far set is the 7 larger-scale runs of Sec. 4.1. We will state this rule in Sec. 4.1.
  • Code snippet. At the end of the appendix, we will add a self-contained snippet that rebuilds the splits and folds and fits all three laws with our exact procedure. It reproduces the fits of the three laws in Tables 1, 3 and 5 on the public data.

We cannot release our full code base, the SK-Grid losses or the checkpoints. Instead, the snippet reproduces the main public-data results, and App. E documents the SK-Grid training recipe.

Thank you again: this check fixed a real error in our baseline and improve the quality of the paper. We hope the corrected comparison, the explicit target and the code snippet address your concerns.

Thank you for revisiting the Farseer fits in Table 1. I look forward to seeing the corrected results and discussion in the revised paper.

For Table 3, could you clarify how you selected the training runs for the three scaling laws after holding out the highest-compute runs on each D/N slice?

I was a little surprised by how much the gap between Skaling and Farseer varies across datasets for far extrapolation from the L-shaped grid. On SK-Grid, the MAPE is 1.15% for Skaling and 10.79% for Farseer. On the Farseer dataset, the two are much closer, at 1.51% and 2.35%, respectively.

Reviewer_4ngEOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work takes a closer look at the classic Chinchilla scaling law formulation, proposing a slightly updated version (dubbed Skaling law), that introduces an additional exponent. The proposed law aims to correct for model size and training data interactions that lead Chinchilla to over- or underestimate the loss in the extremes (with low data or overtraining). The authors provide compelling, extensive experiments to support their claim.

Critical strengths
  1. The conducted experiments are extensive, and cover a useful range of model and data sizes. I especially appreciate including extrapolation experiments both on the whole N-D grid (as described in Secs. 3.2 and 3.3.) and along 1D iso-ratio lines (as described in Sec. 3.4). 
  2. The proposed, more efficient “L-shape” sampling scheme is compelling and could be useful to the community.
  3. The proposed scaling law is simple, intuitive, and recovers both Kaplan's and Chinchilla's scaling laws as special cases.
  4. I appreciate the model-free data analysis that precedes and grounds the proposed updated scaling law.
  5. The paper is very clearly written and easy to follow, and includes comprehensive experimental details.
  6. The results in App. A.3 on the compute-optimal ratio varying with compute are interesting.
Critical weaknesses
  1. In Tab. 2 “SK-Grid, L-Shape”, Chinchilla and Skaling seem to hit the upper bounds for parameters A and B. Where does that come from? Is this an issue with the fitting, or is the scaling law poorly constrained here?
  2. I find it curious that, when removing $E$ in App. F, Chinchilla actually seems to outperform Skaling on Farseer, if I understand correctly. To me, this seems to indicate that the mis-estimation of E in Chinchilla scaling laws (and potentially also in Skaling) plays a larger role than stated in the main paper, and that the missing N-D-coupling might not be the only salient issue in Chinchilla? Could you elaborate on this comparison?
  3. I find the very late mention of Busbridge et al. that proposed this exact scaling law somewhat disingenuous. While I appreciate the in-depth analysis of the proposed scaling law and the comprehensive experiment to understand its benefits, and I think this work makes a compelling contribution, large portions of the main text seem to be written as if this formulation is entirely novel (also giving it a new name). Since the authors are aware of this formulation being used effectively before, I think the original work should be credited from the start, and the authors should consider naming the law after that work.
  4. The 1D per-ratio power laws (Tab. 3) suffer from trying to fit the laws on relatively few data points. If I understand correctly, the setting with the most data points (Chinchilla-optimal) is also the setting where Skaling doesn't perform well. It's also unclear whether the differences in fit are statistically significant here.

 

Nits:

  • L62 “we study the Skaling law” is misleading, if the author propose this
  • L101 remove “But, “ or “, however,”
  • The authors jump straight into analyzing the data in Sec. 2, without explaining further where this data comes from; which model is trained using what pipeline. I understand that this is introduced in Sec. 4.1, but at least a reference to that would be helpful.
Questions for authors
  1. I'd be interested whether Skaling would fit some of the data provided in [1], and generally how the authors view the connection between that work and this.

 

[1] Language models scale reliably with over-training and on downstream tasks https://arxiv.org/pdf/2403.08540

3 comments

Thank you for the detailed and constructive review. We will (i) credit Busbridge et al. from the introduction on, and (ii) add a new analysis on the data of [1] (comment 2/2), where Skaling improves over-training extrapolation.

W1. Parameters at their bounds (Table 2, SK-Grid L-shape). Thanks for catching this: AA is at its upper bound for both laws. We refitted both with the bound on AA and BB raised to 103010^{30} (MAPE %, mean over 5 folds):

SK-Grid, L-shapeInterp.Ext. NExt. DFar
Chinchilla, bounded2.196.093.6314.63
Chinchilla, unbounded2.457.293.9516.51
Skaling, bounded0.330.770.551.15
Skaling, unbounded0.491.220.902.91

Both fits get worse and more variable across folds, but the conclusion holds: Skaling is more accurate in every regime, by over 5× in far extrapolation. It is not an optimizer failure: the fit is poorly constrained in one direction, for a different reason in each law:

  • Skaling. AA and α\alpha trade off, so AA can grow with little effect on the fit. Unbounded, AA rises to 8×1078 \times 10^{7} and α\alpha to 0.82, but the effective exponents (Sec. 4.2) barely move: kαk\alpha = 0.23 and kβk\beta = 0.18 (0.24 and 0.20 bounded).
  • Chinchilla. On the L-shape, it distorts its size exponent to absorb the interaction it cannot represent (Sec. 4.2): α\alpha goes from 0.34 (full grid) to 0.52 (L-shape), and to 0.63 unbounded.

We will add the unbounded fits to App. C and flag the bound hits in Table 2.

W2. Dominated-pair fitting and EE (App. F). Your reading is correct, and EE is hard to identify at these scales for both laws (Sec. 4.2). App. F tried to cope with this, but it mostly reweights the grid:

  • What it does. Each run enters the objective once per dominated pair it belongs to. Runs in the grid's corners (large models on few tokens, small models on many) belong to few pairs, so they count less: on Farseer, runs with D/N<2D/N < 2 or >300> 300 are 20% of the data but get 12% of the weight. These corners are where Chinchilla errs most (Fig. 1).
  • Effect. Chinchilla gains on the far runs but loses on larger DD (1.98 → 2.61%), whose held-out runs are mostly small models on many tokens (D/N>100D/N > 100).
  • Overall. This is the only one of 16 cells (two grids, full and L-shape, four regimes) where Chinchilla, with either fit, beats Skaling. On the L-shapes, its far error stays 5–11× higher.

So EE is a real issue, but removing it does not make up for the missing interaction. We will add this explanation to App. F.

W3. Credit to Busbridge et al. We agree. We found Busbridge et al. late in the writing and only added it to the related work. We will:

  • state in the introduction and Sec. 3.1 that the form was introduced by Busbridge et al. (2025), for the supervised losses of their distillation scaling law;
  • rewrite the phrasing that presents the form as ours (e.g., L62).

On the name: it refers to the Kaplan–Chinchilla bridge that Busbridge et al. themselves point out ("Hoffmann et al. (2022) use γ = 1, whereas Kaplan et al. (2020) use β = 1"), and a short label helps in a paper that compares several forms. We will introduce it as "the Skaling form of Busbridge et al. (2025)", so the credit is attached to the name.

W4. Per-ratio power laws (Table 3). You are right, and this is what Table 3 shows:

  • With many points, a per-ratio power law works well. It sees only the cheap runs of its own slice. In the Chinchilla-optimal band, slices have 14–17 of them, and it matches Skaling (0.77 vs 0.88%, within its slice-to-slice spread of ±0.39).
  • With few points, a law in NN and DD has the edge. Elsewhere, slices have as few as 3 cheap runs for 3 parameters. A law in NN and DD is fitted on all slices at once, so runs at other ratios help it extrapolate along a thin slice. On undertrained slices, Skaling and Farseer reach 0.45% and 0.61% (corrected Farseer fit, see our reply to qWJd), against 1.32% for the per-ratio law.
  • But only if its form fits the whole surface. Chinchilla, fitted on the same pooled runs, gains little: it is worse than the per-ratio law in two of the three groups (1.52% undertrained, 3.47% Chinchilla-optimal) and overall (2.34 vs 1.45%).

So in its best case, the per-ratio law is competitive with Skaling, and where runs are scarce, the laws that model the NN–DD interaction are clearly better. A lab rarely has 15 cheap runs at every ratio it may train at, and only a law in NN and DD says how to trade one for the other. We will detail this claim in Sec. 4.3.

Nits. We will rewrite L62 (see W3), remove the doubled "But, … however," on L101, and add the missing pointer from Sec. 2 to Sec. 4.1 for the Farseer runs.

AuthorsOpenReview ↗

Skaling on the data of [1]

Thank you for this suggestion. We fitted Skaling to the runs released with [1] and will add the analysis to the appendix and discuss [1] in the related work.

How the two laws relate. [1] starts from Chinchilla with tied exponents (α=β\alpha = \beta) and rewrites it in terms of compute C=6NDC = 6ND and token multiplier M=D/NM = D/N:

L=E+(aMη+bM−η)C−η,η=α/2.L = E + \left(a M^{\eta} + b M^{-\eta}\right) C^{-\eta}, \qquad \eta = \alpha/2.

Under the same assumption, Skaling becomes

L=E+(a′Mα/2+b′M−α/2)kC−αk/2.L = E + \left(a' M^{\alpha/2} + b' M^{-\alpha/2}\right)^{k} C^{-\alpha k/2}.

This has three consequences:

  • k=1k = 1 recovers [1]'s law exactly.
  • For any kk, the compute exponent does not depend on MM, so [1]'s parallel lines are preserved.
  • kk decouples the shape of the over-training penalty (the MM-dependent factor) from the compute exponent. In [1]'s law, both are set by η\eta.

We therefore expect Skaling to help when extrapolating across token multipliers. When extrapolating in compute at fixed MM, both laws reduce to a power law in CC, so we expect them to match. We see the two works as complementary: [1] shows that over-trained models scale reliably, and Skaling adds the N–D interaction that matters across MM.

Setup. [1] released runs of four small models (11M–411M) trained with MM = 10–640 tokens per parameter, plus 1.4B and 6.9B models. Each model is trained on one of three corpora: C4, RedPajama or RefinedWeb. As in [1] (App. F), we drop the off-trend M=5M = 5 runs. For each model we predict two losses: the loss on C4 validation text, which [1] uses for every model, and the loss on held-out text from the model's own training corpus (its Paloma split). Three corpora × two losses give 6 settings. Fitting is identical for all laws and uses the setup of the paper (App. C). The tied variants (α=β\alpha = \beta) compare Skaling with [1] like-for-like, with kk as the only difference. The untied variants compare Skaling with Chinchilla.

Error on held-out runs (mean absolute relative error, %). Each cell gives the error on the C4 loss / on the own-corpus loss, each averaged over the three training corpora.

ExtrapolationFit → predict[1]'s lawSkaling (α=β\alpha = \beta)ChinchillaSkaling
Over-training16 runs, M≤80M \le 80 → runs with M≥160M \ge 1601.85 / 2.451.30 / 1.921.61 / 2.161.20 / 2.39
Model size21 runs, 11M–154M → 411M0.90 / 1.950.86 / 2.052.21 / 2.812.53 / 2.95
To 1.4B and 6.9B, [1]'s default fit5 runs ([1]'s Table 1) → 1.4B, 6.9B1.11 / 5.381.07 / 5.381.39 / 5.51–
To 1.4B and 6.9B, larger fit25 runs: 11M–411M at M≤320M \le 320, plus 1.4B at M=20M = 20 → 1.4B at the largest MM, 6.9B1.32 / 4.252.17 / 5.531.35 / 4.101.45 / 4.31
  • Over-training: on the C4 loss, Skaling lowers the error by about 30% relative to [1]'s law (tied) and by 25% relative to Chinchilla (untied). On the own-corpus loss, the tied variant improves by 22%; the untied one does not (2.39 vs 2.16).
  • Model size and [1]'s default fit: Skaling is on par with [1]'s law, as expected at fixed MM. [1]'s default fit uses only 5 runs, too few to identify kk (and too few for the 6 parameters of untied Skaling). Both untied laws extrapolate poorly in model size here: 11M–411M is too narrow a range to estimate a separate size exponent, and tying acts as a regularizer.
  • Larger fit (25 runs): tied Skaling's gap to [1]'s law comes mostly from the models trained on C4 (5.1 / 5.7% vs 2.7 / 3.0%). According to the released configurations, 4 of the 25 C4 runs (154M at MM = 160, 320; 411M at MM = 80, 320) were trained on a different copy of C4. They lie ~0.05 nats below the trend, and because they are among the highest-compute points, they pull the more flexible fit down at large scale. Excluding them from every law's fit gives 1.55 vs 1.38% (C4 loss) and 4.52 vs 4.53% (own-corpus loss), and the over-training gain remains (1.21 vs 1.82%).
  • The high own-corpus errors in the last two rows come from a single target, the RedPajama 1.4B model at MM = 640, which every law misses by 11–16%.

[1]'s data show the same coupling. Fitted on all small runs, kk is 0.4–0.7 in 5 of the 6 settings, tied or untied, far from the additive k=1k = 1. The exception is RedPajama with the own-corpus loss, where k≈1k \approx 1.

I thank the authors for the comprehensive reply. I think the described changes to the fitting and the early-mention of Busbridge et al. largely address my main concerns.

I also find the in-depth analysis of [1] very interesting, thanks for including this!

Overall, I think this is an interesting paper and I recommend acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.