acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Pre-pretraining

27 views

Abstract

We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in compute-matched experiments, across four PPT curricula (-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes: data-constrained, data-unconstrained, and a high token-to-parameter-ratio regime (881M parameters trained on 50B tokens). When natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum on natural-language validation loss, with no consistent downstream accuracy advantage from PPT. We find that broader PPT gains reported in prior work can be attributed to the weight initialization scheme. In an NCA reproduction, we recover a 5.4% PPT advantage under the original initialization; changing the initialization improves both methods (the no-PPT baseline by 9.0%), leaving them indistinguishable. However, when PPT compute is *additive* to, rather than *competing* with, the natural-language budget, PPT gains persist even after improving the baseline initialization at small natural-language budgets, but vanish by 1B natural-language tokens. Finally, knowledge acquired during PPT fails to survive subsequent natural-language training; interleaving the same data throughout training preserves it, though it does not significantly improve performance. Our results suggest limited benefit from synthetic pre-pretraining as a distinct training stage and motivate rethinking how such synthetic data are used in language model training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pre-pretraining, Synthetic data, Language model pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AXvKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores the effectiveness of Pre Pre training. It shows that some of the gains in prior work can be recovered from different initialisation schemes, and that the PPT data may be better suited to using natural data, opposed to using that generated by rules (ie. formal languages or neural cellular automata). They also explore interleaving this data throughout training, and show benefits in the retention of performance on the PPT data when training happens on PT data.

The paper addresses an important question. However, I believe the conclusions are not yet supported sufficiently, given the lack of hyperparameter reoptimisation in figure 2 and uncertantity about the number of gradient steps the models use in PPT, which significantly affect the computational structures acquired during PPT.

Critical strengths

Comprehensive evaluation: The paper evaluates PPT across several regimes (data constrained, data unconstrained, and beyond chinchilla training*) . These experimental settings and considerations are valuable contributions. *(The claim that over-chinchilla has not ben explored by prior PPT work should be clarified, see Cheng et al. 2026, who train a 254M model on 100B natural language tokens)

Distributing the PPT data across training: An interesting setting that has received little exploration in prior work, which has mainly focussed on frontloading PPT data. This setting shows benefits in retention of PPT performance in comparison to frontloading. However, its unclear how relevant this is in regards to the original motivation for PPT, which is not necessarily performing well on PPT tasks, but to initialise useful computational structures that can be adapted for further training on natural data. For example, Jiang et al. (2026) isolate useful computations in individual components and find that these can outperform transferring the whole model from PPT to PT. Nonetheless, this is a valuable contribution that motivates further exploration of data schedules for integrating alternative data sources such as NCA.

  • Cheng et al. (2026), Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
  • Jiang et al. (2026), Procedural Pretraining: Warming Up Language Models with Abstract Data.
Critical weaknesses

Additive Compute Setup and Baseline: Firstly, the authors reuse the setting reported in Lee et. al (2026), learning rate, number of NCA tokens, weight decay etc. However when changing the initialisation there is no guarantee this is the optimal configuration or even a good configuration. The authors do acknowledge this by stating "this experiment is meant to show how much performance gain is available on the baseline before any PPT data is introduced,". The experiment shows this - improving random initialisation by changing the statistics of the distribution. But this experiment is not strong enough for the abstract claim that "PPT gains reported in prior work can be attributed to the weight initlilzation" and is not supported by this experiment. To make this claim it would be required to at sweep the hyperparameters that Lee et al. sweep. In the PPT phase, Lee at al. state they sweep various batch sizes, learning rates, and weight decays. Secondly, a useful comparison is missing: that of using Lee et al. setting in PPT then using the reinit to the 0.02 of the embeddings for PT. This would isolate if the new setup for PPT is not a good choice and this experiment would clarify it. Thirdly in Figure 5 2/3 initialisation schemes find NCA to be superior to the no PPT baseline (even without retuning the ppt phase). This makes the abstracts claims of "PPT gains reported in prior work can be attributed to the weight initlilzation" unsupported.

Substantially larger PPT batch sizes then prior work: The authors use significantly larger batch sizes then the prior work they are comparing with. Jiang et al. (2026) use ~1k to 8k tokens per step for their language experiments, Lee at al. (2026) ~16k and Hu et al. (2025) ~64k. In contrast this paper frequently uses batches of 256k, 512k or 1M tokens. For a fixed token budget, these larger batches will result in significantly less gradient updates during PPT. I am concerned that the models may not have received enough gradient steps to acquire the underlying structre of the PPT data that previous work has found to be very beneficial for subsequent PT. Furthermore, the authors do no report PPT task performance and the end of PPT. It is difficult to distinguish whether PPT fails to give transferable benefits or if the setup is insufficient for the PPT data to be learnt sufficiently in the first place.

  • Lee at al. (2026), Training Language via Neural Cellular Automata
  • Hu et al. (2025), Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
Questions for authors

Figure 2:

  • Given that Lee at al. (2026) explicitly state their method tunes learning rate, batch size and weight decay during PPT on NCA, could the authors investigate if retuning this hyperparameters when changing the initialisation restores the benefits of NCA. It seems particularly relevant given figure 5 shows NCA retaining 4.4% perplexity improvement under the spectral initialisation
  • Could the authors investigate retaining the weights learned during PPT using Lee at al.'s original initialisation, while initialising the new embeddings etc at the start of PT using the alternative initialisation? This would clarify if the disappearance of NCA's advantage is really due to a better initialisation for PT or changes to what is acquired from training on NCA.

Batch sizes

  • Please clarify if smaller batch sizes, closer to the prior work was explored during hyperparameter optimisation. If not what is the justification for making this change?
  • Please clarify the number of gradient steps performed during PPT and the performance for the experiments. It is unclear whether the models receive enough optimisation steps to acquire useful computational structure.
5 comments
AuthorsOpenReview ↗

Extension of Reproduction Experiment & Larger batch size justification

First of all, thank you for an engaging and thorough review. Here we address the points you raise:

Tuning hyperparameters of the reproduction experiment. The purpose of the NCA reproduction was to (1) demonstrate how sensitive PPT is to weight-initialization, a factor prior literature hasn’t explored (2) that such sensitivity extends beyond our own experimental settings (Figure 8), i.e. OpenWeb text, versus our FineWeb, and Llama versus our Nanochat. That being said, we have taken your suggestion and tuned the PPT and PT learning rates (lr) of this reproduction, each until the optimum is bracketed. We first tune the PT lr, then given this optimum, tune the PPT lr. Under the NCA’s optimal lr, at three seeds, we find the original 5.4% gain still largely collapses to ~0.7%, and hence tuning the hyperparameters for the NCA reproduction does not restore its 5.4% benefit.

PT LR (at PPT lr 1e-4)5e-4 (paper’s)1e-32e-33e-34e-3
NCA16.0615.4915.2315.1615.52
PPT LR (at PT lr 3e-3)5e-51e-4 (paper’s)2e-44e-4
NCA15.2315.1615.2415.58
PT LR5e-4 (paper’s)3e-3 (NCA opt.)
No PPT16.10±0.0215.27±0.05
NCA16.07±0.0315.16±0.02
C4 unified-15.20±0.05
C4 separate-15.56±0.07

Furthermore, we reproduce the C4-PPT control, both with the optimizer reset at the PPT-to-PT boundary, as in prior work ([1], [2], [3]), and with a unified optimizer across both phases, as in our work. The reset version is worse than the no-PPT baseline, as also observed in Figure 3 of [3], while the unified version matches NCA-PPT, further supporting the findings in our own setup. Note that only NCA's lr was tuned here, and we reuse it for the no-PPT and C4 runs. Despite these results, we agree that the wording "attributed to" was too strong, and have changed it to "sensitive to the weight initialization scheme" (line 17).

Initialization in the reproduction experiment. Thank you for raising this point. We want to clarify that we do run the experiment in the way you describe it. In Lee et al.’s NCA PPT setup, the token embeddings are initialized from a N(0,0.02) during the PPT phase, but at the transition to the PT phase, the token embedding table is instead initialized from a N(0,1). In our experiments we leave the PPT phase untouched, and change N(0,1) to N(0,0.02) at the transition. We now describe this with greater clarity in the Appendix B.

Larger batch sizes than prior PPT work. We too had observed that our hyperparameter search selected PPT batch sizes larger than those used in prior work. We double-checked this choice at the time and found no clear improvement from smaller batches. We now reconfirm this with additional experiments in the NCA 1% data-unconstrained setting, at the 1e18 and 3e18 scales. Here, we scale the PPT lr found at our 1M batch size to each smaller batch size with the square-root scaling rule [4], and run at three seeds per batch size.

PPT batch1e183e18
16k0.9804±0.00060.9254±0.0005
32k0.9808±0.00040.9259±0.0006
64k0.9807±0.00010.9266±0.0010
1M0.9803±0.00050.9239±0.0001

As we can see, reducing the PPT batch size does not improve final validation BPB at either scale, and none of these settings brings NCA closer to the no-PPT baseline (0.9794 and 0.9234).

What are the PPT step counts? Below are the average PPT and PT steps per experiment. Prior work uses ~100 to 5,000 PPT steps (2M–164M tokens) ([1], [2], [3]). Most of our step counts fall within this range. The lower counts come from the data-constrained and 1% PPT experiments, which are different settings to those in prior work.

SettingPPT tokensPPT stepsPT steps
Data-constrained
1% PPT3.1M81,052
5% PPT16M301,021
10% PPT25M81835
Data-unconstrained
1% PPT, 1e1811M263,676
1% PPT, 3e1816M385,619
1% PPT, 9e1831M7610,799
5% PPT, 1e1855M1883,393
5% PPT, 3e1885M2965,579
5% PPT, 9e18150M4589,971
Over-Chinchilla
1% PPT500M476189,463
Interleave
5% PPT, separate phase258M78818,504

Why interleaving? The interleaving suggestion is made given the forgetting on the PPT distribution observed in Figure 4. We observe that what the PPT models learn about their own distribution is forgotten after the phase boundary. We have added more discussion on this in Section 8.

Again, thank you for your comments, we believe our manuscript has been improved because of your review.

[1] Hu et al. Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases. https://arxiv.org/abs/2502.19249

[2] Jiang et al. Procedural Pretraining: Warming Up Language Models with Abstract Data. https://arxiv.org/abs/2601.21725

[3] Lee et al. Training Language Models via Neural Cellular Automata. https://arxiv.org/abs/2603.10055

[4] Malladi et al. On the SDEs and Scaling Rules for Adaptive Gradient Algorithms. https://arxiv.org/abs/2205.10287

Dear Reviewer,

Just following up to summarize our response above:

  1. Confirmed our reproduction experiment is conducted in the way you reccomended it to be conducted in.
  2. Hyperparameter tuned the reproduction experiment - showed our claims hold.
  3. Provided additional compute matched experiments in the reproduction setting - strengthening the validity of the compute matched claims we make in the paper.
  4. Reducing the PPT batch size does not significantly improve PPT results.
  5. Report the PPT step counts are mostly within range of prior PPT work.

Thank you again for your review, we’d truly appreciate your thoughts on our responses, as we believe it strengthened the manuscript.

Thank you for the engagement, response and experiments.

These new results do strengthen the paper, however I remain concerned that the PPT setup has not been sufficiently optimised and searched through, including weight decay, duration etc. . It also remains unclear whether the models sufficiently learn the PPT tasks before transitioning to language pretraining.

I therefore maintain my original score.

Reviewer_oR8eOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors provide a careful empirical study of pre-pretraining (PPT), which introduces procedurally generated synthetic data as a phase 0 before usual pretraining on natural language text. Prior works had reported improvements in natural language validation loss as a result of pre-pretraining. This work considers a wider variety of settings, including an additive compute setting in which one is allowed to include a PPT phase at the start for free, and a compute-optimal setting, where PPT FLOPs trade off with PT FLOPs. Within the compute-optimal setting, they consider three subsettings: compute-constrained, data-constrained, and Chinchilla over-trained (which essentially amount to three different points across the spectrum of over-trained to under-trained). Overall, the authors recapitulate earlier findings that PPT can help at small scales in the additive setting, but uniformly find that it fails when PPT FLOPs trade-off with PT FLOPs.

Critical strengths

This paper provides a carefully executed empirical study of pre-pretraining across several well-controlled settings, including a compute-controlled setting where PPT compute trades off with PT compute. I found this work very refreshing in that it does not necessarily aim to make a methodological contribution, but instead very carefully executes hyperparameter tuning to provide solid evidence that an early phase of PPT does not help at sufficiently large compute scales. I found all of the experimental setups (e.g., compute-constrained, data-constrained, Chinchilla-overtrained) to be very carefully thought out, and I was quite convinced by the paper's negative result.

Critical weaknesses
  • The only serious concern I have with the experimental methodology is in the data-constrained setting: see my question below.
  • As a somewhat more minor concern, I do feel that the writing sounds fairly AI-generated throughout, and I think the paper would benefit from the authors taking a few passes and smoothing out the language.
Questions for authors

In the data-constrained setting, it appears that you tune over parameter count, which suggests you also tune over epochs. Do you also tune over weight decay, following Kim et al. [1]? This is known to drastically change takeaways.

[1] Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto. Pre-training under infinite compute. https://arxiv.org/abs/2509.14786

3 comments

Thank you for this review. We appreciate your highlighting of the careful execution and breadth of our empirical study. We aimed to rigorously explore the several types of compute settings where PPT could be useful in, and ultimately which settings its limitations lie within. We have updated the paper with more discussion on both these points (see Section 8). We have also improved the writing throughout since the last version. With regards to your question:

Do we tune weight decay? We do not tune weight decay. Our data constrained experiments operate under a fixed compute budget rather than the unlimited-compute regime of Kim et al. (2025). Kim et al. show that when compute is unlimited and data is very constrained (on the order of 200M tokens), much larger weight decay, together with increased model size and ensembling, makes better use of that compute. Hence, we think the Kim et al. set up is not directly analogous to our synthetic data settings - they use pure natural language and train under infinite compute.

Understood, thanks for the clarification. In general, as one takes more epochs over the train data, regularization becomes increasingly critical---this is well understood from the days of ImageNet and studies such as Kim et. al. have confirmed this to be true for LLM pretraining.

I will maintain my score as the rebuttal doesn't change my opinion of the paper.

Dear Reviewer,

We appreciate your feedback and positive review.

Thank you for your engagement.

Kind regards

Reviewer_wf1qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Earlier work claims language models learn better if they first train briefly on synthetic data- pre-pretraining (PPT). This paper asks whether PPT is a better use of compute than spending the same compute on natural-language text. The authors test up to four PPT corpora under a fixed compute budget in three settings: limited text, plentiful text, and a model trained on far more tokens than is compute-optimal. They find that PPT never gives lower natural-language validation loss than spending that compute on text, even when the text is limited and must be repeated. They also find that some earlier PPT gains shrink once the baseline's weight initialization is changed, and that what the model learns during PPT is largely forgotten once text training starts.

Critical strengths

For each method and budget, the paper searches model size separately (lines 194–195) and tune hyperparameters separately (Appendix). There's a real-text control, and results are averaged over three seeds. Earlier work mostly asked "does adding PPT help?" This asks "is PPT the best use of the same compute?".

Critical weaknesses
  • The results are averaged over three seeds with one-standard-deviation error bars (lines 278–279), and "significant" means these bars don't overlap (e.g., Table 15 caption). No formal statistical test is used. With only three seeds this is a loose check, so small real gains from PPT could go undetected. -The abstract says PPT gains in prior work "can be attributed to the weight initialization scheme" (line 17), but it also notes that in the additive setting "PPT gains persist even after improving the baseline initialization at small natural-language budgets" (lines 21–23). So initialization explains only part of the earlier gains, and the first claim is stated too broadly.
Questions for authors

How many passes over the 100M natural-language tokens does the data-constrained setting make (lines 229–230)? Could you add a harsher setting with many more passes, for example fewer unique tokens at the same budget? If PPT still loses there, the "never" claim becomes more stronger.

5 comments
AuthorsOpenReview ↗

Epochs under data constrained setting & Significance of reported metrics

Thank you for this review. With regards to your questions and concerns:

How many epochs in the data-constrained setting? This setting fixes total compute and unique data to 100M tokens. 1.96–2.78 epochs are used in this experiment. These values are a result of hyperparameter optimization to ensure each PPT method, and baseline get their respective optimal model size (561M–708M) given the compute and data constraints. If we were to simply fix epochs, model size, and thus total compute, conclusions could conflate the baseline overfitting the data, with the effect of PPT. In such case, part of the total compute would be used for PPT, so the PPT methods would see fewer epochs in this regime than the baseline. By allowing model size to be a free variable, while constraining compute, the optimal number of epochs can be chosen, isolating PPT’s effect.

Weight initialization claim is too strong. Thank you for this point. We agree that the wording "attributed to initialization" was too strong, and so we have changed it to "sensitive to the weight initialization scheme" (see line 17).

Why no data constrained experiment at 50M budget? The goal of this experiment is to explore whether, under a very small amount of natural language data with fixed total compute and epoching allowed, does PPT help? We chose 100M tokens as it is a standard budget in data-constrained settings ([1], [2]). We don’t expect for the conclusions to differ at a lower budget in this regime. That being said, we do study this scarcer 50M budget in another one of our settings - the additive-compute experiments (Section 4).

Significance of reported metrics. We appreciate your point on this. In the paper we report the mean and standard deviation of every benchmark we evaluate, which is in line with previous PPT work ([3]). We have now checked our results using standard error, and find our conclusions hold. In terms of validation BPB, our primary metric, no result changes. Using standard error to compare methods only separates the baseline more from other synthetic PPT methods. However, on a few downstream benchmarks, the tighter intervals flip some results. Those are listed below.

BenchmarkSettingMethodScore
DCLM COREData-unconstrained, 1e18No PPT0.088±0.004
k-Dyck 5%0.098±0.002
Best C4-PPT0.106±0.002
Data-unconstrained, 3e18No PPT0.121±0.009
NCA 5%0.133±0.002
k-Dyck 5%0.135±0.004
Best C4-PPT0.137±0.003
BLiMPData-unconstrained, 3e18No PPT0.780±0.002
k-Dyck 5%0.783±0.001
Best C4-PPT0.780±0.003
Data-unconstrained, 9e18No PPT0.794±0.002
k-Dyck 1%0.798±0.001
Best C4-PPT0.794±0.005
Data-constrained, 1e18No PPT0.721±0.002
NCA 10%0.730±0.004
k-Dyck 5%0.724±0.001
k-Dyck 10%0.728±0.004
DeclRef 1%0.727±0.002

However, the same phenomenon we mention in the paper occurs (caption of Table 1): the PPT methods still do not separate from the C4-PPT baseline. This means the PPT methods do not significantly improve over a natural language baseline. Thus, this analysis does not change how we read the benchmark results. We still claim that we see no consistent downstream accuracy advantage from PPT. Thank you for bringing this to our attention, we will add the standard error caveat and this table to the appendix in the final revision.

[1] Warstadt et al. Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora. https://aclanthology.org/2023.conll-babylm.1/

[2] Vegesna et al. NanoGPT Slowrun: Language Modeling with Limited Data and Infinite Compute. https://github.com/qlabs-eng/slowrun

[3] Lee et al. Training Language Models via Neural Cellular Automata. https://arxiv.org/abs/2603.10055

Dear Reviewer,

To summarize our response, upon your recommendation we have:

  1. Changed the wording of the initialization claim
  2. Analyzed the results under standard error - and still find that our claims hold.

Thank you again for your review, we’d truly appreciate your thoughts on our responses, as we believe it strengthened the manuscript.

Dear Reviewer,

As the discussion period ends today we would greatly appreciate the chance to discuss your review and our responses with you.

Thank you again for your review,

Kind regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.