acceptodds
Under review as a conference paper at ICLR 2027

Small-Scale Experiments: Are We There Yet?

Abstract

Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver. Instead, researchers have found them unreliable at small scales, starting at 4M parameters, and concluded that sizable models cannot be avoided. We show this is not the case and that the confounding factor is hyperparameter tuning. In fact, small models are highly sensitive, but hyperparameter sensitivity fades with scale. This small-scale sensitivity makes scaling laws easy to miss because they only emerge on the fully tuned frontier, and reaching that frontier requires an extensive search far beyond what most ever run. By ablating the basic scaling law recipe, we show well-tuned hyperparameters matter more than any other ingredient. Further, we reveal why those hyperparameters become easier to find: as scale increases, the hyperparameter loss surface becomes lower dimensional. Nevertheless while scaling laws exist in small models, extrapolation hits statistical limitations. A holistic approach is required. Synthesizing our insights with the recent literature, we develop a new methodology for model-centric research and demonstrate it on a question that once took the field years to settle: where to place normalization layers in the transformer architecture. From small-scale experiments, we recover the large scale result: pre-normalization works better as models grow in size. With the right tools and a better understanding, small-scale experiments can deliver on scaling laws' long-awaited promise.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling-law, optimisation, hyperparameter, LLM

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_1mSaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why scaling laws estimated from small language models can be unreliable and argues that insufficient hyperparameter tuning is a major cause.

It characterizes how hyperparameter sensitivity changes with scale and develops a methodology for small scale experiments, which is evaluated on Llama style language models and a pre norm versus post norm case study.

Critical strengths

The paper performs an extensive hyperparameter study across a wide range of model and data scales, including held out larger models for evaluating extrapolation.

The paper goes beyond showing that tuning matters and analyzes how hyperparameter sensitivity changes with scale, then connects these observations to a practical methodology for small scale experiments.

Critical weaknesses

The main findings are based on one Llama style model family, one pretraining dataset, and one optimizer setup, so it is unclear how broadly the observed decrease in hyperparameter sensitivity transfers to other training settings.

The estimated effective number of hyperparameters decreases with scale, but the noisy quadratic fits use different portions of the score distribution at different scales. It is therefore unclear whether this trend is robust to the choice of fitting threshold.

Questions for authors

The noisy quadratic fits use roughly the best 7 to 8% of configurations at the smallest scales, but up to 50% at the largest scales. Can the authors report whether the estimated decrease in effective hyperparameter dimension remains when using the same tail fraction across scales, or provide a sensitivity analysis over a common range of thresholds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1mSa

Thank you for the positive review and for your precise and actionable questions. Following your suggestions, we ran the threshold analysis you proposed with about 900 configurations per scale from 4M to 537M, and new experiments with a different dataset and a different model family. We respond to your question first, then address your general concerns.

Q1: Effective number of hyperparameters at a common threshold

Thank you for this insightful question. Following your suggestion, we repeated the analysis with common thresholds across scales, using the final training loss of about 900 configurations per scale from 4M to 537M.

Our findings in Section 4.1 give a good explanation for why one might set a higher quantile for the threshold in larger models: as models scale, a larger fraction of hyperparameter configurations fall near the optimum. Imagine instead that we defined the threshold in terms of how suboptimal a config is, then the quantile must naturally increase. For example, the 10th percentile at 4M parameters is 0.21 BPC worse than the optimum. At 537M parameters, the 10th percentile only extends .03 BPC above the optimum: 2.5 times the seed noise (σ≈0.012\sigma \approx 0.012 BPC). This lower signal-to-noise ratio can make the distribution more challenging to estimate.This is why our original percentiles grew with scale.

Below, we fix the threshold in terms of the degree of suboptimality. We show γ\gamma for this common window above the best loss:

Design, window (bits per character)4M8M17M34M67M134M268M537M
Ladder, 0.1522222111
Ladder, 0.2022222111
Ladder, 0.2522211111
Ladder, 0.3022211111
Random, 0.1523212121
Random, 0.2023222111
Random, 0.2523222111
Random, 0.3023221111

With a common loss threshold, the effective number of hyperparameters still drops from two to one as models grow, for both designs.

Independently of any threshold, sensitivity also decreases strongly with scale. Using the same runs, below are the share of configurations within 0.1 BPC of the best and the gap between the 25th-percentile and best score, all in bits per character (BPC):

Scale4M8M17M34M67M134M268M537M
Within 0.1 of best5.5%3.9%7.4%11.9%13.3%20.5%24.8%30.4%
25th percentile − best0.500.540.450.290.240.140.100.08

We see that more runs fall near the optimum as the scale increases indeed.

One model family, dataset and optimizer

Our study is model-centric, so we deliberately kept the dataset fixed while changing the model. Following your suggestion, we ran two new sets of experiments with the same 128 hyperparameter configurations at every scale from 4M to 134M, using (1) the same model family on a different dataset, DCLM-baseline [1], and (2) a different model family on the same dataset, a recurrent architecture built from Hawk's RG-LRU blocks [2]. Below is the gap between the 25th-percentile and best final training loss (nats), and the share of configurations within 0.1 nats of the best:

Gap4M17M34M134M
Llama, FineWeb-Edu (paper)1.391.410.720.47
Llama, DCLM (new)1.231.290.670.43
Hawk, FineWeb-Edu (new)1.010.640.420.31
Within 0.1 of best4M17M34M134M
Llama, FineWeb-Edu (paper)1.6%2.4%4.7%5.5%
Llama, DCLM (new)1.6%2.4%6.2%5.5%
Hawk, FineWeb-Edu (new)1.6%3.1%3.9%7.8%

In both new settings, sensitivity decreases with scale as in our main study. The gap shrinks about 3 times from 4M to 134M, and the share of near-optimal configurations grows 3 to 5 times.

Thank you again for your feedback, which made our analysis more precise and broadened our experiments. We included these additional results in the revised manuscript. We made a considerable effort to address all of your concerns, and believe the new results substantially strengthen our paper; we hope that you would consider raising your score.


References

[1] Li, Jeffrey, et al. "Datacomp-lm: In search of the next generation of training sets for language models." Advances in Neural Information Processing Systems 37 (2024): 14200-14282.

[2] De, Soham, et al. "Griffin: Mixing gated linear recurrences with local attention for efficient language models." arXiv preprint arXiv:2402.19427 (2024).

Thank you for the detailed rebuttal and additional experiments, which substantially address my original concerns. I will therefore raise my rating from 3 to 4.

Thank you for raising your score! Your feedback made our analysis more precise and broadened our experiments, and we included the new results in the revised manuscript. We appreciate your helpful and constructive review.

Reviewer_cvg2OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper shows that past failure in getting scaling laws working at the 'small scale' is down to hyperparameter optimization. This happened because

  1. well-tuned models are required for a good scaling law fit (the paper shows empirically that it is sufficient - increasing the hparam trials at each scale does reduces train val and test mse of the scaling law fit).
  2. it is more challening to tune smaller sclae models (in both the data and model size sense), as measured my the intrinsic dimensionality of the hyperparameter optimum

The paper suggests as a check for well-tuned hyperparameter tuning that the score distribution approaches a noisy quadratic.

Using each of these findings, the paper presents a recipe for hparam tuning at the very small scale (which is extremely cheap due to ~quadratic growth in compute cost with model size for well-trained models) in order to minimize the uncertainty on training extrapolations from experiments upon a test point of interest.

The paper further applies the findings to demonstrate the practical utility of pre-norm over post-norm design as a case-study of their recipe.

Critical strengths

1. The paper resolves an outstanding problem in the the scaling toolbox, coupled with "practical theory", deep insights, and a followable recipe for future researchers to use.

The paper is very clearly written, has carefully designed experiments that carefully test specific hypotheses, and the practical take-aways that anyone thinking about scaling and hyperparameters can (and should) immediately start using. The access to scaling studies at a smaller scale significantly increases the community that can run scaling studies.

Critical weaknesses

1. One critical component of "the modern scaling recipe" is width/depth reparameterization, i.e. hyperparameter transfer, is missing from the discussion.

Methods like MuP, Complete(d)P [1,2,3] allow one to tune at one scale (model size/data/batch) and transfer optimal hyperparameters to another scale. It would be good to have one of these methods as a baseline hyperparameter allocator.

Further, "in theory" if you are tuned well enough and transfer works well enough, the hyperparameter search budget could be used entirely on the smallest model, with no searches needed at any larger model, which would give even better extrapolation uncertainty per total computational budget.

[1] G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, J. Gao. "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer." arXiv:2203.03466, 2022. https://arxiv.org/abs/2203.03466

[2] N. Dey, B. C. Zhang, L. Noci, M. Li, B. Bordelon, S. Bergsma, C. Pehlevan, B. Hanin, J. Hestness. "Don't be lazy: CompleteP enables compute-efficient deep transformers." arXiv:2505.01618, 2025. https://arxiv.org/abs/2505.01618

[3] B. Mlodozeniec, P. Ablin, L. Béthune, D. Busbridge, M. Klein, J. Ramapuram, M. Cuturi. "Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration." arXiv:2512.22382, 2025. https://arxiv.org/abs/2512.22382

Questions for authors

1. [Clarification] What is a hyperparameter?

The idea that scaling laws fit much better for optimal hyperparameters than non-optimal raises a challenge for me. One could imagine that the model type itself (e.g. Transformer vs RNN vs SSM) or the optimizer type (e.g. AdamW vs Muon) are part of the hyperparameter space. Taking the paper's finding at face value, we would conclude that some optimizer-architectures combinations would have lower scaling law extrapolation uncertainty than other combinations? Is that the case? Or is that some class of hyperparameters give disconnected regions on the hyperparameter manifold? Or should block type and optimizer type not really be considered hyperparameters in the sense the paper deals with?

2. How do hyperparameter transfer methods interact with your recommendation?

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cvg2

Thank you for the positive assessment, and for finding that our paper resolves an outstanding problem in the scaling toolbox with practical take-aways. Inspired by your feedback, we ran new μ\muP experiments at widths 128 to 2048, showing that our findings hold under hyperparameter transfer and inform how to use it. We address your points below.

Hyperparameter transfer

Thank you for this great point; hyperparameter transfer is a very interesting direction. Our focus is on the properties of the loss surface at each scale and how they change with scale, rather than on tuning hyperparameters across scales, so our work complements transfer methods such as μ\muP. Since we ran our experiments with standard parametrization, a natural question is whether our findings still hold under μ\muP.

Our results are invariant to reparametrizations at a single scale. At a fixed scale, a transfer rule reparametrizes the hyperparameters through a smooth, invertible map. At the optimum, the Hessian becomes J−⊤HJ−1J^{-\top} H J^{-1}, with JJ the Jacobian of this map, so its rank, the effective number of hyperparameters γ\gamma, is unchanged, and near-optimal volumes only change by a constant factor. In addition to this theoretical argument we tested the parts of μ\muP that change the model itself (initialization and output multipliers) empirically.

Smaller models remain more sensitive under μ\muP. Following your suggestion, we implemented μ\muP (base width 128) and trained the same 256 random configurations at five widths (depth 4, 8M–313M parameters). We report the gap between the 25th-percentile and best final training loss, and the share of configurations within 0.1 nats of the best:

Width12825651210242048
Gap (nats)1.641.280.780.480.32
Share within 0.1 nats2.3%3.9%4.3%5.9%9.4%

From width 128 to 2048, the gap shrinks 5 times and the near-optimal share quadruples, as in standard parametrization. Therefore, our findings reflect properties of the loss surface which are invariant to these kinds of reparametrizations.

Small-scale tuning determines how well μ\muP transfers. We search MM random configurations at width 128, transfer the best one to width 1024 without further tuning, and report its loss above the best of the 256 configurations at width 1024:

MM (configurations at width 128)8163264128
Loss above best at width 1024 (nats)0.290.160.100.060.04

Searching the small scale more thoroughly makes the transferred configuration 7 times closer to the optimum. Since small models are the most sensitive, transferring good hyperparameters requires more thorough search at smaller scales, and our diagnostics show when a search was sufficient. Thus, our findings help practitioners use μ\muP better. We updated the manuscript to reflect this initial exploration of the interaction between hyperparameter transfer methods and our findings.

Spending the search budget at the smallest scale

We agree that with good enough tuning and transfer, the search budget could be concentrated on the smallest models. Our results address the first condition, since our diagnostics tell when a small scale is well tuned. We updated the manuscript to state that our compute-allocation analysis assumes independent searches at each scale, and that integrating it with transfer methods is an exciting direction for future work.

Question 1: What is a hyperparameter?

This is a great question. It helps to separate two parts of extrapolation error. The approximation (bias) is how well the functional form approximates the true compute–loss surface, and the estimation (variance) is how reliably we can find parameters close to the best approximation. Better hyperparameters reduce the bias. In the same way, the optimizer and architecture change the compute–loss surface itself and therefore the quality of scaling laws as an approximation. One intuition is that as an algorithm gets better at extracting information from data, it becomes constrained mainly by data and parameters, so better optimizers and architectures may have cleaner scaling laws.

In our framework, discrete choices such as the optimizer or architecture type are handled with separate quadratics (Section 2), i.e., disconnected regions of the loss surface, each with its own sensitivity and scaling law. So yes, different architecture–optimizer combinations can have different extrapolation uncertainty, as illustrated by our pre-norm vs. post-norm case study. Post-norm (widely considered the worse architecture) was harder to tune and produced worse scaling laws. In contrast, pre-norm (the better architecture) produced better scaling laws, just as you predict.

Thank you again for your thoughtful feedback and support. These additions connect our findings to the modern scaling recipe, and we included them in the revised manuscript.

Thank you for the detailed responses and additional experiments, in particular the MuP results.

Table 10 is particularly interesting, and seems to also support spending the search budget at the smallest scale when parameterized for transfer.

My assessment is unchanged, and I am keeping my Rating at 4. Thank you for a great paper; the practical recipe will be useful to many people running scaling studies.

Thank you for your kind words and for your thoughtful review, which helped us connect our findings to hyperparameter transfer. We're excited for the community to build on this work and we hope practitioners find it a helpful guide for designing and properly tuning small-scale experiments!

Reviewer_2912OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper investigates the problem: Can small-scale experiments (roughly below 100M parameters according to the paper's context) be used to establish reliable scaling laws? If so, how?

The paper's main findings and conclusions are:

  1. First, (power-law) regularities can emerge at very small scales if hyperparameters are sufficiently tuned. The paper further proposes using NQL as a diagnostic method, and provides mechanism interpretation that smaller models are more sensitive to hyperparameters than larger ones with dimension fitting and random-search loss density as tools.
  2. Second, although regularities exist at small scales, their long-range extrapolation is not reliable.
  3. Third, the paper advocates a holistic approach, a suite of diagnostics beyond only the scaling law, to leverage small-scale experiments. A case study of Pre-Norm vs. Post-Norm selection is provided.

In summary, the first finding demonstrates that small-scale experiments can establish scaling laws, and the latter two address how to reliably use them.

Critical strengths

The paper provides convincing empirical evidence that hyperparameter tuning quality is a critical factor in revealing small-scale scaling regularities (i.e., the first finding discussed above). The main evidence is presented in Section 3, where Figures 1 and 3 exclude other scaling-law fitting methodologies as the main influencer, and Figure 3 directly demonstrates the substantial impact of hyperparameter tuning.

Critical weaknesses

The paper provides convincing evidence that sufficient hyperparameter tuning can reveal small-scale scaling regularities (Section 3) and enable extrapolation over a certain range (to 268M). However, its broader claim that the proposed methodology enables small-scale experiments to reliably guide large-scale model development is not adequately supported. There remains a key gap between the presented results and reliable model-selection decisions at scale.

Section 5 demonstrates that long-range extrapolation can be suspicious. To address this, the authors propose a holistic approach incorporating several complementary diagnostics. However, these diagnostics (tuning adequacy, hyperparameter sensitivity, and perplexity–capability correspondence) do not directly resolve the long-range extrapolation uncertainty; nor, if regarded as a suite of decision toolkits, does the paper establish a clear decision rule for combining them when different scaling-law fits lead to conflicting predictions.

The Pre-Norm vs. Post-Norm case study is a concerning example. In Figure 21, separate and joint fitting produce different long-range predictions. The authors reason with the first three diagnostics to favor Pre-Norm and trust the joint fitting, while it actually yields a substantially worse validation MSE for Post-Norm than separate fitting. There is no clear rule about which fitting should be trusted or how diagnostics should be weighted. The final decision still relies on considerable subjective judgment rather than a validated, reproducible selection procedure.

Consequently, the paper does not sufficiently demonstrate that the proposed holistic methodology enables reliable large-scale model selection.

Questions for authors
  1. Could the authors clarify how the validation scales (67M and 134M) were used in selecting the scaling-law estimation methodology?

  2. Section 5.2 attributes the instability of long-range extrapolation primarily to inaccurate estimation of the irreducible loss term, while this estimation error seems could be reduced with more observations in principle. Therefore, it is unclear whether the observed instability reflects a finite-sample estimation problem or a more fundamental obstacle. Could the authors investigate whether increasing the number of hyperparameter configurations (e.g., 256, 512, or more) reduces the variability of the estimated scaling laws?

  3. The largest held-out model in this paper has only 268M parameters, while prior scaling-law studies typically validate predictions at larger scales (e.g., Porian et al. [65], Gadre et al. [23], Bhagia et al. [5]). Could the authors provide more direct evidence that their proposed holistic approach reliably guides model selection at larger scales, as raised in the weakness above? If larger-scale experiments show contrasting evidence, solidly establishing these limitations for small-scale experiments would itself constitute a valuable finding, rather than making the current claim.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2912, 1/2

Thank you for acknowledging the strengths of our work and for providing helpful, actionable feedback. Inspired by your feedback, we ran additional experiments at significantly larger scales (up to 4B parameters, 16× our previous largest held-out model) and with more configurations per scale (up to 832), demonstrating that our findings generalize beyond the scales we covered in our submission. We respond to your questions first then address your general concerns.

Q1: How the validation scales were used

We split the scales into training (4M–34M), validation (67M–134M) and test (268M). In our research, we fit the models on train then used validation to explore different results. We reserved test for the final evaluations you see in the paper.

Q2: Finite-sample estimation or fundamental obstacle

Following your suggestion, we bootstrapped the fit on 4M–34M, drawing NN configurations per scale with replacement and refitting Equation 1 to the best loss per model size and token budget. Below is the standard deviation across resamples of the fitted irreducible error ϵ\epsilon and of the predicted 1B loss at its final token budget:

Configurations per scale8128512832
Std. of ϵ\epsilon (BPC)0.390.260.210.17
Std. of predicted 1B loss (BPC)0.2220.0650.0230.017

From 8 to 832 configurations, the standard deviation drops 2.3 times for ϵ\epsilon and 13 times for the 1B prediction; fitting more scales also reduces the standard deviation of ϵ\epsilon (0.37 for 4M–17M vs. 0.22 for 4M–67M, at 128 configurations). As the sample size (number of configurations) increases, the error drops; however, it drops for the irreducible error very slowly because it is difficult to estimate. This confirms our argument that "extrapolation eventually reflects the irreducible error, a quantity whose estimates vary greatly across different samples", and that these estimates improve with more samples. Thus, the instability is a practical rather than a fundamental obstacle. The scaling law’s functional form is accurate, but that functional form is statistically difficult to extrapolate. You can reduce the error by exactly what our paper recommends: tuning more thoroughly at small scales; however, as we also point out: distant extrapolation may require more tuning than is practically possible. We included these results and clarified this point in Section 5.2.

Q3: Evidence at larger scales

Based on your feedback, we extended the held-out scales to 537M, 1B, 2B and 4B. Our findings generalize: the better tuned the small scales, the more reliable the large-scale predictions. Repeating the bootstrap with NN configurations per scale at 4M–67M (100 resamples), the standard deviation of the predicted loss at each held-out scale, as % of the actual loss, is:

NN537M1B2B4B
819.5%23.7%27.5%29.7%
1283.4%4.6%5.8%7.0%
2562.1%2.6%3.2%4.0%
5121.6%1.9%2.3%3.0%

From 8 to 512 configurations, the 4B prediction becomes 10 times more precise: small-scale tuning pays off up to 60 times beyond the largest fitted model.

We also show that the remaining error shrinks as the fit spans more scales. Below is the relative error (actual − predicted)/actual of all-configuration fits:

Fit537M1B2B4B
4M–34M−8.8%−11.8%−15.0%−18.5%
4M–134M−2.7%−4.2%−6.2%−8.8%
4M–268M–−1.4%−2.7%−5.0%
4M–537M–−0.5%−1.6%−3.6%

Fits spanning six or more scales stay within 4.2% up to 8 times beyond the largest fitted model and within 9% at 16–32 times, and all fits slightly underestimate how good larger models become. We now state these results explicitly in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 2912, 2/2

The role of the joint fitting in the pre-/post-layer norm experiments

We believe the point you raised regarding the joint fitting reflects a miscommunication of our reasoning when exploring this direction. The joint fitting is not a core part of our methodology or diagnostic steps. We intended it to address the irreducible error: since ϵ\epsilon is hard to estimate from small scales, tying it across the two architectures trades variance for bias and provides a sanity check for the comparison. It only appears in Section 5.3 where it reveals how the irreducible error term drives the difference between the scaling laws. We do not claim that the joint law’s extrapolation should be trusted on its own, and its higher validation MSE for post-norm is a legitimate observation. We understand how the current way we wrote our paper might present the joint fitting in a confusing way, so inspired by your feedback, we updated the manuscript accordingly to move Figure 21 (left) to the main text, and properly contextualize the joint fitting and Figure 21 (right) in the appendix as an exploration for future work that aims to address the issues with estimating the irreducible error. We also revised the caption and conclusion.

We also want to note that our conclusion that pre-norm is the better choice does not rest on Figure 21a alone. It follows from all the diagnostics together, which is the holistic approach we advocate for. In particular, post-norm was harder to tune at every turn and pre-norm scales better near the data where scaling laws are most reliable, rather than far from it where our paper shows extrapolation is unreliable.

The main purpose of our work

We would like to reiterate the main purpose of our work. We do not propose a single decision rule but a set of diagnostics for what can go wrong. Our paper shows that small-scale experiments might look unreliable when in fact they are simply under-tuned. For example, Xiao (2024)[1] concluded that small batches train language models poorly, which Marek et al. (2025, Figure 5)[2] traced to leaving Adam's β2\beta_2 untuned. At scale, our new configuration ablations show that tuning small scales more thoroughly makes large-scale predictions far more reliable: from 8 to 512 configurations per scale, the spread of the 4B prediction shrinks 10x (Q3). With well-tuned small scales, fits spanning six or more scales stay within 9% up to 4B. Thoroughly tuned small-scale experiments therefore remain informative up to 4B parameters, and we quantify how far they can be trusted.

We also now cover in our paper the relationship between our findings and hyperparameter transfer methods. Please see our response to Reviewer cvg2 for a thorough description.

Thank you again for your constructive feedback, which helped us increase the scope of our experiments and communicate our message more clearly. We made a considerable effort to address all of your concerns, and believe the new results substantially strengthen our paper; we hope that you would consider raising your score.


References

[1] Xiao, Lechao. "Rethinking conventional wisdom in machine learning: From generalization to scaling." arXiv preprint arXiv:2409.15156 (2024).

[2] Marek, Martin, et al. "Small batch size training for language models: When vanilla sgd works, and why gradient accumulation is wasteful." Advances in Neural Information Processing Systems 38 (2026): 148837-148862.

AuthorsOpenReview ↗

Follow-up as the discussion period nears its end and summary of additional results

Thank you again for your helpful feedback. As the discussion period nears its end, we wanted to follow up on our response and summarize the other updates we made to the paper following the other reviewers' feedback.

Hyperparameter transfer. Following Reviewer cvg2, we trained the same 256 random configurations with μ\muP at widths 128 to 2048. Smaller models remain more sensitive under μ\muP, as in standard parametrization. The gap between the 25th-percentile and best loss shrinks 5 times and the share of configurations within 0.1 nats of the best quadruples. Searching the small scale more thoroughly also makes the transferred configuration 7 times closer to the optimum, so our findings help practitioners use μ\muP better.

Effective number of hyperparameters at a common threshold. Following Reviewer 1mSa's suggestion, we repeated our analysis with a common loss threshold across scales, using about 900 configurations per scale from 4M to 537M. The effective number of hyperparameters still drops from two to one as models grow.

Other datasets and model families. Following Reviewer 1mSa's suggestion, we ran the same 128 configurations at every scale from 4M to 134M on a different dataset, DCLM-baseline, and with a different model family built from Hawk's RG-LRU blocks. In both settings, hyperparameter sensitivity decreases with scale as in our original experiments.

We included all of these results in the revised manuscript. We also hope that the experiments we ran in response to your questions, at scales up to 4B parameters, were helpful, and that our response cleared up the confusion about the joint fitting in the pre-norm vs. post-norm case study. Please let us know if you have any other questions. We would appreciate it if you could revise your assessment in light of these revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.