acceptodds
Under review as a conference paper at ICLR 2027

How Model Growth, Boundary Operators, and Recursion Improve Scaling Laws

Abstract

Scaling laws predict how loss decreases as computation increases. Yet how architecture affects scaling laws is rarely examined. While a better constant brings scale-invariant compute-efficiency gains, a better exponent leads to power-law improvements in performance as computation increases. In this work, we comprehensively study how model growth, recursion, and boundary operators affect scaling coefficients. Specifically, we run more than 30 distinct scaling ladders over compute budgets of approximately to FLOPs. We find that better architectures can improve the *exponents*. In particular, our best architectural variant, *Untied-Grow*, achieves a compute-efficiency gain at FLOPs, and the trend extrapolates to FLOPs, where our model matches GPT-3 13B on CORE. We find that weight sharing is suboptimal in fresh-token training. However, recursion has a useful regularizing effect in data-constrained, multi-epoch training. In this regime, we find that it is compute-optimal to increase the number of loops with scale, outperforming model-size scaling even with tuned weight decay. To understand these gains, we examine computational depth and find that exponent improvements correlate with faster growth in computational depth.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scaling laws, Deep Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6yAdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the scaling laws of model growth, recursion, and boundary operators in a looped transformer. With compute budget of 10^18 to 10^20 flops in 30 runs, the authors show that model growth and boundary operators provide compute-efficiency gains that increase with scale and increasing the loop count with scale in the data-constrained setting is better than scaling model sizes or tuning weight decay. A main insight is that architectural changes can, despite claim in other works, yield improvements to scaling exponents rather than just the constants of the scaling laws

Critical strengths
  1. This work covers a range of architectures of looped transformers and shows the improvements caused by architectures
  2. The experiment recipe tuning covers the major aspects of scaling hyperparameters
  3. The insights gained are important, and showcase the importance of considering architectural details when considering the scaling laws of models.
Critical weaknesses
  1. The discussion of overfitting in the variants of looped transformers is not convincing enough. Appendix D provides supporting evidence through experiments with data repetition, weight decay, and weight sharing. But these mainly show final validation losses, so it is still unclear whether looping reduces overfitting or simply helps optimization. Showing training and validation loss over time would help distinguish the two.

  2. The authors are presenting the plots using a linear scale in the y axis. This is different than most other works in this space. Is there are a reason for this choice? It makes reasoning about their work needlessly challenging compared to the rest of the literature.

Questions for authors
  1. Could you please make the plots in the log-log scale? If not, what is the reason of using linear scale for the y axis?
  2. Could you please provide evidence that overfitting is avoided? [See the weakness point 1 above]
  3. What are the irreducible floors E for each corpus in your fitting results? Are they consistent in the fitting between variants of different architectures?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 6yAd

Thank you for your thoughtful questions. Inspired by your comments, we conducted additional analyses and report the results below.

Overfitting effects

Motivated by your questions, we examine how model depth and looping affect the generalization gap and training loss.

DepthLoop-1Loop-2Loop-3Loop-4Loop-6Loop-8
60.8910.8670.8480.8450.834—
81.4711.4241.4041.3741.3631.393
102.1982.1192.0692.0832.0572.020
123.0402.9452.8992.8622.8132.798
143.9533.8353.7553.7243.6843.679

Table 1: Generalization gap (validation loss minus training loss) across depths and loops. Models are trained on 100M unique tokens for 10 epochs with WD = 0.05. Results are similar when WD=0.8.

DepthLoop-1Loop-2Loop-3Loop-4Loop-6Loop-8
62.9492.9252.9162.9102.909—
82.4932.4802.4752.4842.4872.447
102.0282.0332.0472.0102.0192.050
121.5471.5431.5511.5641.5921.598
141.0451.0691.0911.1031.1181.122

Table 2: Training loss across model depths and loops under the same training conditions as Table 1.

Increasing depth lowers training loss but widens the generalization gap. Increasing loops, by contrast, generally narrows the gap. At larger depths, looping slightly increases training loss while reducing the generalization gap. In particular, at depth 14, increasing looping from 1 to 8 raises training loss from 1.045 to 1.122 while reducing the generalization gap from 3.953 to 3.679. As a result, the overall validation loss decreases from 4.998 to 4.801. This supports a regularizing effect of looping and helps explain its benefits in the data-constrained setting. We will incorporate this analysis into the revision.

Logarithmic Y-Axis

Thank you for pointing this out. The current plots use a linear y-axis. We will switch to a logarithmic y-axis in the revision. Because the plotted losses span a narrow range, this change has only a modest effect on their visual appearance.

For the discussion of irreducible loss, please see “Exponent Analysis” in our response to Reviewer Hvrs.

Thank you again for your supportive and constructive suggestions. We hope these clarifications and additional analyses, inspired by your comments, help answer your questions. We would appreciate if you could consider raising your score in light of our response.

Dear Reviewer 6yAd,

Thank you again for your feedback. We made a strong effort in our response, including new analyses and paper edits. As the discussion phase approaches its end, we are happy to answer any additional questions you may have or provide further clarifications. We would greatly appreciate if you would increase your score in light of our response.

Reviewer_HvrsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how boundary operators, weight sharing, model growth, and model shape affect empirical language-model scaling. It tunes architecture-specific training recipes and compares scaling ladders spanning roughly 10^18–10^20 FLOPs. Boundary operators and growth improve fitted compute-loss exponents, while untying mostly improves constants in fresh-token training. Untied-Grow reaches a reported 1.55x compute multiplier at 10^20 FLOPs, with an additional 7.4B run testing extrapolation. A separate repeated-data study finds that increasing tied loop count is more efficient than scaling model size under the evaluated ten-epoch setting. KL effective-depth measurements provide a tentative interpretation.

Critical strengths
  1. The controlled comparisons are the main strength. Shape, boundary operators, tying, and growth are often changed together in recursion studies; this paper separates them and tunes the training recipe for each family. Reporting interpolation-based compute multipliers alongside fitted exponents also helps distinguish observed efficiency gains from conclusions that depend on the loss-floor fit.

  2. The contrast between fresh-token and repeated-data training is informative. The loop-scaling advantage survives weight-decay tuning of the model-size baseline, and the larger held-out run provides a useful extrapolation check. These results give concrete guidance about when weight sharing is helpful, rather than a general claim that recursion is always better.

Critical weaknesses
  1. The reported regression errors quantify a narrower source of uncertainty than is needed for the exponent comparison. Section 3.2 and Appendix A.3 estimate the loss floor from Vanilla and then hold it fixed when fitting other families. The resulting standard errors do not include uncertainty in that floor, recipe selection, or training variation. Several intervention effects are only 0.002–0.003 in the exponent, so sensitivity to these choices matters. The interpolation-based efficiency gains remain valid evidence of an in-range improvement; the stronger claim about persistent exponent differences needs a more complete uncertainty analysis.

  2. The larger run tests the Untied-Grow extrapolation, but not both sides of the claimed efficiency gap. Figure 3 has a measured 7.4B Untied-Grow point and an extrapolated Vanilla baseline at that budget. The widening comparative advantage, and especially the multiplier at 10^25 FLOPs, should therefore be described as predictions. The main repeated-data optimum is conditional on a fixed 100M-token, ten-epoch grid. Appendix D additionally compares fresh data and a 250M-token pool repeated four times; these useful controls still do not jointly optimize epoch count, model size, and recurrence.

Questions for authors
  1. How stable are the exponent differences when the shared floor is refitted or perturbed and the smallest budgets are omitted? Please separate the reported regression errors from training-run variation. This can largely be addressed by reanalyzing the existing runs.

  2. Is there a measured Vanilla result near 1.23e21 FLOPs? If not, please make clear which comparative multipliers are extrapolated and provide uncertainty for those predictions.

  3. Please delimit the repeated-data optimality claim to the evaluated regimes, acknowledging the four-epoch controls in Appendix D. Is epoch count jointly optimized anywhere? Please also clarify whether the compute multipliers count training runs alone or include recipe tuning.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer Hvrs (1/2)

Thank you for your thoughtful review and suggestions. Inspired by your comments, we conducted further scaling-law analyses and additional experiments in the data-constrained setting.

Exponent Analysis

To examine how the choice of irreducible loss affects our exponent comparisons, we fit each architecture independently using L(C)=E+A(C/C0)−γL(C) = E + A (C/C_0)^{-\gamma}. The table below reports the fitted coefficients and compares the RMSE with our paper’s approach, which shares Vanilla's irreducible loss EE.

VariablesVanillaUntied-GrowLoop-GrowUntied-2Loop-2Operator-1Deep VanillaDeep Vanilla Grow
EE1.70641.47361.53951.60451.59851.74451.68851.6211
AA1.62551.80681.76801.69161.71391.56551.62821.6819
γ\gamma0.11120.09770.10250.10550.10550.11780.10980.1065
RMSE0.001700.000740.002790.001860.002700.003200.002000.00237
RMSE (paper)0.001700.003970.003530.002510.002920.003320.002030.00283

Table 1: Independently fitted scaling-law coefficients and fitting errors. "RMSE (paper)" uses Vanilla’s fitted irreducible loss as the shared EE, while fitting AA and γ\gamma separately for each architecture.

Independently fitted coefficients give different EE and γ\gamma for different architectures. Vanilla has a larger exponent than Untied-Grow while Untied-Grow has a lower irreducible loss. To interpret this interaction, we note that

−dLdlog⁡C=γ(L−E).-\frac{dL}{d\log C}=\gamma(L-E).

At a given loss (LL) and local slope (−dLdlog⁡C-\frac{dL}{d\log C}), a lower fitted EE corresponds to a smaller γ\gamma. Comparing exponents alone does not fully characterize how loss decreases as a function of compute. Ultimately, we want a faster decreasing loss and a larger compute saving.

In the paper, we treat compute multipliers as the primary metric and exponents are more like a quantity that helps describe how compute efficiency changes (L259-261). To see how a shared EE gives a direct connection between compute efficiency and exponents, suppose we have two architectures M (method) and R (reference). Let CC denote R's compute, and define compute multipliers CM(C)=C/CMCM(C)=C/C_{M}, where CMC_M is the compute for M to match R's loss. With a shared EE, we have

dlog⁡CM(C)dlog⁡C=1−γRγM.\frac{d\log CM(C)}{d\log C}=1-\frac{\gamma_R}{\gamma_M}.

Thus, a larger exponent is equivalent to a faster growing compute multiplier. We will include this quantitative connection in the revision. We will also report exponent ratios in the revision. To be consistent with the existing presentation, we continue to use exponent difference (Δγ\Delta \gamma) in this response.

AuthorsOpenReview ↗

Author Response to Reviewer Hvrs (2/2)

Exponent Analysis (cont'd)

Following your suggestion, we next examine the fitted exponent improvement using different shared values of EE, taken from each architecture’s independent fit, and after excluding the smallest runs.

Shared EUntied-GrowLoop-GrowUntied-2Loop-2Operator-1Deep VanillaDeep Vanilla Grow
Vanilla (paper)+0.006+0.005+0.003+0.003+0.0030.000+0.003
Untied-Grow+0.003+0.003+0.002+0.002+0.0020.000+0.001
Loop-Grow+0.004+0.004+0.002+0.002+0.0020.000+0.002
Untied-2+0.004+0.004+0.002+0.002+0.0020.000+0.002
Loop-2+0.004+0.004+0.002+0.002+0.0020.000+0.002
Operator-1+0.006+0.005+0.003+0.003+0.0030.000+0.003
Deep Vanilla+0.005+0.005+0.003+0.003+0.0030.000+0.002
Deep Vanilla Grow+0.004+0.004+0.002+0.003+0.0030.000+0.002

Table 2: Exponent improvements over Vanilla under different shared EE. Each row uses the fitted EE of the indicated architecture. All exponent standard deviations are below 0.001, except one at 0.0011.

The magnitude of the exponent improvement depends on the shared EE, but the qualitative pattern remains consistent across all values tested. The pairwise comparisons in Table 1 of the paper also remain consistent. For example, using Untied-Grow’s fitted EE, growth increases the exponent by approximately 0.001 for each of the Untied, Loop, and Deep Vanilla architectures.

FilterUntied-GrowLoop-GrowUntied-2Loop-2Operator-1Deep VanillaDeep Vanilla Grow
d≥6 (paper)+0.006+0.005+0.003+0.003+0.0030.000+0.003
d≥8+0.005+0.004+0.002+0.003+0.0020.000+0.002
d≥10+0.004+0.005+0.002+0.003+0.0020.000+0.002
d≥12+0.004+0.004+0.002+0.003+0.0020.000+0.002

Table 3: Exponent improvements over Vanilla after excluding the smallest runs. The filters progressively remove d6, d8, and d10, where d denotes depth

Excluding the smallest runs has a minor effect on exponent improvements.

We use one random seed per configuration, so the reported regression uncertainties do not capture variability across independent training runs.

Data-Constrained Optimality

You are right that we didn't optimize epochs. We chose 10 epochs to study a regime with substantial overfitting. We will explain the setup more explicitly in the revision.

To further assess scalability, we increase the unique corpus while keeping epoch count fixed. This holds the number of repetitions constant as the data budget grows [1]. We therefore run additional experiments on 250M unique tokens for 10 epochs. Please refer to “Scalability of Data-Constrained Results” in our response to Reviewer yKre.

All compute multipliers count training compute only, and exclude recipe-tuning costs. At 1.23×10211.23 \times 10^{21} FLOPs, the Untied-Grow result is measured, whereas the Vanilla comparison is extrapolated. The multiplier at 102510^{25} FLOPs is also an extrapolated prediction. We will label these explicitly in the revision.

Thank you again for your constructive feedback. We put a significant effort into our response, including new experiments and analyses inspired by your comments. We would appreciate if you could consider raising your score in light of our response.

References

[1] Microsoft. (2026). Frontier Pre-training at Microsoft AI

Dear Reviewer Hvrs,

Thank you again for your feedback. We made a strong effort in our response, including new experiments and extensive analyses. As the discussion phase approaches its end, we are happy to answer any additional questions you may have or provide further clarifications. We would greatly appreciate if you would increase your score in light of our response.

Reviewer_yKreOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how three changes (model growth, recursion, and a boundary operator) affect scaling laws. It trains more than 30 scaling ladders from 10^18 to 10^20 FLOPs, and found better architectures can improve the exponents. The best variant, Untied-Grow, reaches a 1.55x compute-efficiency gain at 10^20 FLOPs. Other useful observations were identified, including: weight sharing is suboptimal in fresh-token training, recursion can function as a useful regularization in data-constrained training, scaling law’s exponent improvements correlate with faster growth in computational depth.

Critical strengths

Rigorous per-architecture recipes: Each architecture gets its own tuning, token-allocation fit, and learning-rate rule. This matters a lot because architecture comparisons are usually confounded by recipe choice.

Critical weaknesses
  • Inference cost is not discussed. The best architecture Untied-Grow combines untied model growth with the boundary operator. The compute multiplier counts only training FLOPs, so the deployment trade-off at matched loss is unclear.
  • The data-constrained results are small-scale. They cover compute up to about 8*10^18 FLOPs and pools of 100M or 250M unique tokens (Figure 20). It is unclear whether the result holds for larger data pools and model sizes.
Questions for authors
  • Batch size and context length are fixed across the ladder (Table 2). Since the critical batch size grows with model scaling-up, conclusions might differ under different batch sizes and contexts. Have you checked how the exponent will change with different batch sizes?
  • Will code, recipes, and the ladder checkpoints be released? Reproducing more than 30 ladders with per-architecture recipes is costly.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer yKre (1/3)

Thank you for your thoughtful review. Inspired by your comments, we conducted additional experiments and discuss their results here.

Total Batch Size and Context Length

In the main experiments, we fix total batch size 524,288 (524K) and context length 2048. We re-run the scaling ladder for Vanilla and Untied-Grow and vary either context length or total batch size. Specifically, we consider the following settings:

  • total batch size: 131K, 262K, 1048K.
  • context length: 512, 1024, 4096.

The optimal hyperparameters may change, so we sweep learning rates for different context lengths and total batch sizes. We found that the optimal LR is constant for different context lengths while the optimal LR is halved for batch sizes 131K and 262K.

After adjusting the learning rate, we run the scaling ladders and re-calculate the compute multipliers and fitted scaling coefficients. The results are shown here.

Total batch sizeΔγ (↑)\Delta\gamma\,(\uparrow)Δlog⁡A (↓)\Delta\log A\,(\downarrow)d8d10d12d14d16
131,072+0.0024-0.0414—1.67×1.72×1.78×1.76×
262,144+0.0052-0.03881.46×1.49×1.58×1.66×1.68×
524,288*+0.0055-0.03001.30×1.33×1.36×1.43×1.48×
1,048,576+0.0086-0.03661.25×1.30×1.36×1.42×1.45×

Table 1: Compute multipliers for different total batch sizes. The d8 multiplier at 131K is omitted because d6 Untied-Grow already achieves a lower loss than d8 Vanilla, placing the matched-loss comparison outside the measured range. "*" marks the default setting in the paper.

Smaller batch sizes show greater improvements in the constant term, whereas larger batch sizes show greater improvements in the scaling exponent. Within the measured range, smaller batches consistently achieve larger compute multipliers—for example, approximately 1.6–1.8× at a batch size of 131K. At larger batch sizes, however, the multipliers increase more strongly with scale. Across batch sizes, Untied-Grow’s compute-efficiency advantage over Vanilla grows with scale.

Context lengthΔγ (↑)\Delta\gamma\,(\uparrow)Δlog⁡A (↓)\Delta\log A\,(\downarrow)D8D10D12D14D16
512+0.0045-0.03041.29×1.36×1.39×1.45×1.46×
1,024+0.0048-0.03411.30×1.37×1.39×1.46×1.49×
2,048*+0.0055-0.03001.30×1.33×1.36×1.43×1.48×
4,096+0.0074-0.03001.26×1.29×1.37×1.38×1.46×

Table 2: Compute multipliers for different context lengths. "*" marks the default setting in the paper.

Improvements in both the constant term and the scaling exponent, as well as the resulting compute multipliers, are consistent across context lengths.

Batch size affects both the magnitude and the scale dependence of Untied-Grow’s compute-efficiency gains. Smaller batches yield larger improvements in the constant term and larger compute multipliers over the measured range, whereas larger batches show greater exponent improvements and more rapidly increasing multipliers. Context length, by comparison, has limited influence on either the scaling improvements or the compute multipliers. These results suggest that the exponent improvement is robust across the batch sizes and context lengths.

AuthorsOpenReview ↗

Author Response to Reviewer yKre (2/3)

Scalability of Data-Constrained Results

Larger Training Compute on 100M Data

To address your concern about FLOP limitations, we extend the data-constrained runs (bottom-right panel of Figure 1 in the paper) to up to 1.35×10191.35 \times 10^{19} FLOPs. We compare the weight-decay-tuned K=1 baseline with the optimal-loop configuration, which reuses the baseline's optimal weight decay.

DepthParametersLossFLOPsCompute Multiplier
d10330M3.48641.60e+181.26×
d12490M3.45312.49e+181.41×
d14730M3.42793.80e+181.68×
d161.0B3.41225.53e+181.99×
d181.4B3.40387.55e+182.19×
d201.8B3.39841.02e+192.60×
d222.4B3.38491.35e+192.49×

Table 3: Compute multipliers from optimal looping on a 100M-token dataset trained for 10 epochs. The first four columns describe the K=1 baseline and the final column reports the factor by which optimal looping reduces compute at matched loss.

Compute multipliers continue to increase with training compute, reaching roughly 2.5x around 1.35×10191.35\times 10^{19} FLOPs. Optimal looping continues to outperform baseline. Both methods start to plateau at larger scale, so we estimate the irreducible loss for both of them.

CurveE95% CI
Baseline3.3563[3.3483, 3.3639]
Opt. Loop3.3308[3.3173, 3.3407]

Table 4: Fitted irreducible loss for K=1 baseline and the optimal looping configuration. EE is the irreducible loss in scaling laws. Confidence interval (CI) is estimated by bootstrapping.

Optimal looping has a lower fitted irreducible loss than the baseline, with non-overlapping 95% confidence intervals. These results show that looping’s benefits persist at larger compute budgets.

Larger Unique Corpus Size

To test whether the gains extend to a larger corpus, we run additional experiments on 250M unique tokens for 10 epochs, expanding beyond the 100M-token setting. Similarly, we tune weight decays for K=1 baselines and compare them with optimal looping configurations.

DepthParametersLossFLOPsCompute Multiplier
d6120M3.43031.21e+181.00×
d8210M3.31202.31e+181.03×
d10330M3.23784.01e+181.10×
d12490M3.19546.24e+181.23×
d14730M3.16599.52e+181.46×
d161.0B3.15121.39e+191.80×
d181.4B3.13451.89e+192.04×
d201.8B3.12122.57e+192.31×
d222.4B3.11143.39e+192.61×

Table 5: Same setup as Table 3, but with 250M unique tokens and 10 training epochs.

With 250M unique tokens, compute multipliers also increase with scale. At smaller depths, the gains grow more slowly than in the 100M-token setting because loss saturates more slowly on a larger corpus. At larger depths, however, the multipliers catch up to those observed in the 100M-token setting. This is because models at larger depths approach the irreducible loss, which we estimate next.

CurveE95% CI
Baseline3.0699[3.0587, 3.0805]
Opt. Loop2.9903[2.9767, 3.0028]

Table 6: Same setup as Table 4, but with 250M unique tokens and 10 training epochs.

For the 250M corpus, the drop in irreducible loss is around 0.08, larger than the 0.03 drop in the 100M case. This suggests greater potential benefits from looping when the unique corpus increases.

The advantage of looping persists both at larger compute budgets on 100M unique tokens and with a larger corpus of 250M unique tokens. We will evaluate additional corpus sizes to better understand how these patterns evolve as the number of unique tokens increases. We will include these new results and discuss their implications in the revision.

AuthorsOpenReview ↗

Author Response to Reviewer yKre (3/3)

Inference Cost

To assess the inference cost of model growth, we compare Untied-Grow with Vanilla at matched validation loss. We define the inference multiplier as Vanilla’s forward FLOPs per token divided by Untied-Grow’s. Values above 1 indicate lower inference compute for Untied-Grow.

Depthd8d10d12d14d16d18d20
Compute Multiplier1.30×1.33×1.36×1.43×1.48×1.52×1.55×
Inference Multiplier1.10×1.05×1.02×1.04×1.07×1.06×1.06×

Table 7: Training and inference compute multipliers for Untied-Grow at matched validation loss. Depth refers to the Vanilla baseline.

Untied-Grow’s inference-compute advantage is smaller than its training-compute advantage, but remains positive at matched loss, with multipliers of 1.02–1.10× over Vanilla. We will include this analysis in the revision and discuss growth’s increased inference cost relative to the non-growth model as a limitation. We leave potential reductions through distillation and model compression to future work.

We will release code, training recipes, and checkpoints for the main experiments to support reproduction.

We really appreciate your thoughtful review. We put a lot of effort into our response, including new experiments inspired by your comments. We would appreciate if you could consider raising your score in light of our response.

Dear Reviewer yKre,

Thank you again for your feedback. We made a strong effort in our response, including new experiments and paper edits. As the discussion phase approaches its end, we are happy to answer any additional questions you may have or provide further clarifications. We would greatly appreciate if you would increase your score in light of our response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

We thank all reviewers for their feedback and questions. We provide a general response here to summarize the paper’s main findings and the additional experiments and analyses completed in response to reviews. We also would like to highlight the timeliness and significance of this work. We are one of the only works that shows that architectural interventions can actually change a scaling exponents, contrary to conventional wisdom, which means efficiency increases that compound with scale. The practical implications are potentially profound. We additionally provide insights into the mechanisms for changing scaling exponents, such as effective depth.

We appreciate the reviewers’ recognition of our controlled comparisons and independently tuned compute-optimal recipes. Tuning each architecture separately helps us distinguish architectural effects from differences in training recipes. Our findings cover two settings:

Fresh-token training:

An improvement in the scaling constant gives the same proportional compute saving at different scales. An improvement in the exponent means that the saving grows with scale.

  1. Model growth increases depth partway through training. Boundary operators normalize the residual stream and reinject an earlier representation of the input. Both interventions improve scaling exponents.
  2. Untying weights and changing the depth-to-width ratio improve the scaling constant.
  3. Untied-Grow combines growth, boundary operators, and untied weights. It achieves a 1.55× training-compute advantage over Vanilla at matched validation loss at 102010^{20} FLOPs, with the advantage growing with scale.

Data-constrained training:

When data are limited, adding parameters or training for more epochs increases overfitting. Recursion offers another way to use compute. Recursion applies the same layers without adding parameters, reducing validation loss while leaving the optimal weight decay unchanged.

New experiments:

  1. We ran Vanilla and Untied-Grow scaling ladders across different batch sizes and context lengths. The improvements in scaling exponents persist.
  2. We extended the data-constrained experiments to larger models under the original setting (100M unique tokens, ten epochs) and to a larger dataset (250M unique tokens, ten epochs). The looping advantage persists.

Additional analyses and clarifications:

  1. We explain how exponent differences relate to growing compute savings when architectures share the same fitted loss floor. The exponent improvement persists with different floors. The exclusion of the smallest runs has little effect on exponent estimates.
  2. We quantify inference costs at matched loss. Untied-Grow retains a small inference-compute advantage over Vanilla.
  3. We report training and validation losses in the data-constrained setting. Looping lowers validation loss while slightly increasing training loss and narrowing the generalization gap. This supports a regularizing effect of looping.

We put a lot of effort into these additional experiments and paper edits. We hope reviewers will consider them, together with the paper’s significance in the broader context, in their final assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.