- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the interaction between model size and training data in neural scaling laws. It introduces an outer exponent into the additive Chinchilla formulation to capture this interaction while retaining an analytical compute-optimal allocation. The authors evaluate predictive accuracy across several extrapolation settings and propose an L-shaped sampling design to reduce the cost of fitting the scaling law.
Critical strengths
- The paper offers a clear and interpretable perspective. It examines how model size and data interact to explain systematic prediction errors in additive scaling laws, establishing a direct connection between the problem and the proposed formulation.
- It captures coupling through a single additional parameter while preserving an analytical compute-allocation rule, offering a useful balance between expressiveness and simplicity, which makes the main predictive evaluation systematic.
Critical weaknesses
- The novelty of the functional form is limited. Section 5 acknowledges that Busbridge et al. already used the same formulation. The contribution therefore lies primarily in the analysis and experimental design, and the paper should more clearly establish what new understanding it provides beyond the existing formulation.
- The advantage over Chinchilla depends partly on the fitting procedure. In Table 9, dominated-pair fitting reduces Chinchilla’s far-extrapolation error on the full Farseer grid from 2.46% to 0.79%, outperforming both standard Skaling (2.32%) and Skaling with the same fitting modification (1.55%). This suggests that the main comparison does not fully separate the effects of the functional form from those of the fitting procedure.
- Better loss prediction does not necessarily imply better compute allocation. The far-extrapolation evaluations contain only seven and three additional runs, respectively, and the recommended configurations at substantially larger compute budgets are not directly validated. Table 5 also shows smaller and less consistent gains on the original Chinchilla data, leaving the generality of the conclusions uncertain.
Questions for authors
Can the authors directly compare the configurations recommended by different laws at a held-out compute budget, rather than evaluating loss prediction alone?
3 comments
W3. Loss prediction vs. compute allocation. Both laws derive the allocation from their fitted loss surface with the same formula (App. B), so a law that predicts the loss better, inside and beyond the grid, should also place the optimum better. We also check the allocation directly, not only through loss prediction. We do not claim validated recommendations at FLOPs; Fig. 5 only shows how far the two laws diverge. Three points support the allocation:
- Chinchilla's allocation is already wrong inside the grid (App. A.3, Fig. 5). On Farseer, we estimate the compute-optimal frontier from the runs alone, using GP and MLS surrogates with no parametric law. The optimal tokens per parameter decrease with compute ( and ). Fitted on the full grid, Skaling follows this trend (), while Chinchilla predicts the opposite sign (). A law that is wrong on the runs it was fitted on has no reason to be right beyond them. The held-out check in Q confirms this.
- The far sets are small, but they are not the only test. Table 3 evaluates 112 held-out high-compute Farseer runs, where Skaling's error is 3.9× lower than Chinchilla's. The Farseer Ext. N and Ext. D sets hold another 36 and 66 runs.
On Table 5: the gains shrink where the fitted coupling is weak (–), as the nested form predicts. On the Chinchilla data, the two laws are within 0.12 MAPE points of each other. Skaling thus costs little when the coupling is weak and helps when it is strong, and the fitted says which case applies.
Q. Comparing recommended configurations at a held-out budget. Yes. We fit both laws only on the Farseer runs below a compute budget . We then compare their recommendations at FLOPs, the largest budget of the grid, which they never see, with the model-free frontier of App. A.3. Here is the exponent in :
| Laws fitted on runs with | , Skaling | , Chinchilla | at , Skaling | at , Chinchilla |
|---|---|---|---|---|
| (197 runs) | +0.08 | +0.35 | 264 | 1075 |
| (271 runs) | −0.02 | +0.26 | 158 | 588 |
| (314 runs) | −0.07 | +0.20 | 129 | 439 |
| (352 runs) | −0.10 | +0.14 | 114 | 350 |
| Model-free frontier, all 404 runs (GP / MLS) | −0.14 / −0.15 | 85 / 93 |
- Skaling approaches the frontier as grows. Its slope turns negative from , and at it recommends 114 tokens per parameter, against 85–93 for the frontier.
- Chinchilla's slope has the wrong sign at every . It recommends more tokens per parameter as compute grows, the opposite of the data, and overshoots the frontier at by 4–13×.
The extrapolation is short: the grid ends at FLOPs, so is only half a decade above the largest . This does not show that Skaling's allocation still holds at much larger compute, which we do not have the budget to test. But it is direct, held-out evidence that Skaling allocates compute better than Chinchilla. We will add this table to App. A.3.
Dear Reviewer us36,
As the discussion closes today, we would value your feedback on our reply. In short, we addressed your three concerns and your question:
- Novelty (W1). We agree that the form is not new. We will credit Busbridge et al. (2025) in the introduction and in Sec. 3.1, and we listed what the paper adds beyond the form: model-free evidence of the N–D interaction, a controlled comparison on four datasets, the fitted as a guide to when the coupling matters, the L-shape design, and the compute allocation.
- Form vs. fitting procedure (W2). Even with its better App. F fit, Chinchilla beats default Skaling in only 1 of 16 cells. The alternative objective mainly reweights the grid.
- Allocation (W3 and Q). We ran the held-out test you asked for. Fitted only on runs below a compute budget, Skaling's recommended tokens per parameter at FLOPs move toward the model-free frontier as the budget grows (114 vs 85–93). Chinchilla's trend has the wrong sign at every budget and overshoots the frontier by 4–13×. We will add this table to App. A.3.
We would be glad to clarify anything that remains unclear. Thank you again for your review.
Thank you for the careful review. Your three concerns are fair. Below we (1) state what the paper adds beyond Busbridge et al., (2) show that the advantage over Chinchilla holds when Chinchilla uses its best fitting procedure, and (3) check the allocation directly, including at a held-out compute budget.
W1. Novelty of the functional form. We agree that the form is not new. We will credit Busbridge et al. (2025) in the introduction and in Sec. 3.1, not only in Sec. 5. They use this form for the supervised losses inside their distillation scaling law and note that it fits better than tied forms. They do not compare it with other laws under a controlled protocol, nor study the N–D interaction. What our paper adds:
W2. Functional form vs. fitting procedure. In the main comparison, Chinchilla and Skaling share the same optimizer, objective and fitting budget, and both fits converge, so the difference comes from the form. App. F tries an alternative objective for both laws. Even with whichever of its two fits is better, Chinchilla beats default Skaling in only 1 of 16 cells (two main grids, full and L-shape, four regimes): the one you point out, far extrapolation on full Farseer (0.79 vs 2.32%). The pairwise objective mostly reweights the grid: runs in its corners (large models trained on few tokens, small models trained on many tokens) belong to few dominated pairs, so they count less. On Farseer, the runs with D/N<2 or D/N>300 are 20% of the data but get 12% of the weight. These corners are where Chinchilla errs most (Fig. 1), so it gains on the far runs but loses on larger D (1.98 → 2.61%), and on the L-shape its far error stays 5–11× above Skaling's. See also our reply to Reviewer 4ngE (W2).