- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UNSL, a multivariate curve family for neural scaling surfaces that represent regime transition, bottlenecks, irreducible-performance limits, overfitting, and harmful hyperparameter settings. UNSL is evaluated on held-out extrapolation using RMSLE on reused vision and language scaling data.
Critical strengths
-
Multivariate scaling extrapolation is more useful than fitting one-dimensional parameter count curves. The paper evaluates held-out points rather than relying only on visual fit.
-
The appendix reports per-task RMSLEs. Authors also use validation scheme to prevent points from being larger than the validation point in all input dimensions.
Critical weaknesses
-
Most large-scale vision/language evidence is reused from prior studies, while several demonstrations of special behavior use small MLPs, one-epoch training. UNSL do not establish reliability for modern training runs, data mixtures, and architecture families.
-
Reporting the percentage of tasks "won" hides effect size and instability. Since hyperbreak counts and related choices are selected by validation and fitting uses many optimization steps, these omissions are consequential.
-
Limited set of held-out points from the same pre-existing scaling surfaces used for fitting are evaluated. This does not systematically test multiple extrapolation distances. Can you report results under a standardized extrapolation stress test?
Questions for authors
See weaknesses.
Response to Reviewer gpfk
We thank you for the positive and constructive review.
Weakness 1 (reused data; small MLPs). The reused data are modern, large-scale runs across architecture families, and small MLPs are used only in controlled studies. We reused published scaling data on purpose, because those runs could not have been generated or tuned for UNSL, and every form is compared on the same data. The runs cover Transformers, linear RNNs, ViT, MLP-Mixer and ResNets, and they include compute-optimal and constant learning-rate schedules as well as multi-epoch training (scales are in our "General response to all reviewers" comment). The controlled studies each need many full training runs, which would be far more costly at LLM scale. The non-monotone effect of overfitting is also evaluated at scale, in multi-epoch LMs up to 8.7B (Fig. 27) and in JFT vision models (Fig. 17). The benchmark does not vary data-mixture proportions, so the paper makes no claim about them.
Weakness 2 (effect size; instability). The effects are large and consistent, and model selection never sees the held-out points, so it cannot favor UNSL. Effect sizes complement "% of tasks won", and the revision reports them in Sec. 4 (Table 3, also in the general response). The baselines' extrapolation errors are 3.4x (CF), 28x (DC), 2.0x (A1), 1.44x (A2) and 1.18x (A3) UNSL's, where A1-A3 are the paper's ablation functional forms, which are simpler versions of UNSL. All paired p≤0.01.
On instability, we make four points.
Weakness 3 (extrapolation distance). The held-out points span a range of distances, and a controlled study stress-tests extrapolation in several inputs at once. In the benchmark, held-out points sit at the large-scale end of each surface, which matches how scaling laws are used in practice, namely to forecast larger runs of the same setup. Reading off the paper's plots, the 8.7B model is held out entirely, 2x beyond the largest fitted model (Fig. 27, data-constrained LM runs). The largest Chinchilla models are held out, and in the upstream panel the ~16B model is about 2.4x beyond the largest fitted model (Fig. 32, Chinchilla and constant-learning-rate LM runs). The ~7B Transformer and HGRN2 models are fit only on their early training steps and extrapolated to about 6-20x their last fitted step, depending on the model and task (Fig. 32). In the controlled study, UNSL stays accurate an order of magnitude beyond the fitting range (App. H, Fig. 6).
The reused runs are large-scale and could not have been tuned for UNSL, the effect sizes are large (Table 3), the fits are stable, and the held-out data include models at least 2x larger than any fitted one and points about an order of magnitude beyond the fitting range (Figs. 27, 32 and 6). We believe this addresses each weakness the review lists and strengthens the case for acceptance.