- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper explores the effectiveness of Pre Pre training. It shows that some of the gains in prior work can be recovered from different initialisation schemes, and that the PPT data may be better suited to using natural data, opposed to using that generated by rules (ie. formal languages or neural cellular automata). They also explore interleaving this data throughout training, and show benefits in the retention of performance on the PPT data when training happens on PT data.
The paper addresses an important question. However, I believe the conclusions are not yet supported sufficiently, given the lack of hyperparameter reoptimisation in figure 2 and uncertantity about the number of gradient steps the models use in PPT, which significantly affect the computational structures acquired during PPT.
Critical strengths
Comprehensive evaluation: The paper evaluates PPT across several regimes (data constrained, data unconstrained, and beyond chinchilla training*) . These experimental settings and considerations are valuable contributions. *(The claim that over-chinchilla has not ben explored by prior PPT work should be clarified, see Cheng et al. 2026, who train a 254M model on 100B natural language tokens)
Distributing the PPT data across training: An interesting setting that has received little exploration in prior work, which has mainly focussed on frontloading PPT data. This setting shows benefits in retention of PPT performance in comparison to frontloading. However, its unclear how relevant this is in regards to the original motivation for PPT, which is not necessarily performing well on PPT tasks, but to initialise useful computational structures that can be adapted for further training on natural data. For example, Jiang et al. (2026) isolate useful computations in individual components and find that these can outperform transferring the whole model from PPT to PT. Nonetheless, this is a valuable contribution that motivates further exploration of data schedules for integrating alternative data sources such as NCA.
- Cheng et al. (2026), Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
- Jiang et al. (2026), Procedural Pretraining: Warming Up Language Models with Abstract Data.
Critical weaknesses
Additive Compute Setup and Baseline: Firstly, the authors reuse the setting reported in Lee et. al (2026), learning rate, number of NCA tokens, weight decay etc. However when changing the initialisation there is no guarantee this is the optimal configuration or even a good configuration. The authors do acknowledge this by stating "this experiment is meant to show how much performance gain is available on the baseline before any PPT data is introduced,". The experiment shows this - improving random initialisation by changing the statistics of the distribution. But this experiment is not strong enough for the abstract claim that "PPT gains reported in prior work can be attributed to the weight initlilzation" and is not supported by this experiment. To make this claim it would be required to at sweep the hyperparameters that Lee et al. sweep. In the PPT phase, Lee at al. state they sweep various batch sizes, learning rates, and weight decays. Secondly, a useful comparison is missing: that of using Lee et al. setting in PPT then using the reinit to the 0.02 of the embeddings for PT. This would isolate if the new setup for PPT is not a good choice and this experiment would clarify it. Thirdly in Figure 5 2/3 initialisation schemes find NCA to be superior to the no PPT baseline (even without retuning the ppt phase). This makes the abstracts claims of "PPT gains reported in prior work can be attributed to the weight initlilzation" unsupported.
Substantially larger PPT batch sizes then prior work: The authors use significantly larger batch sizes then the prior work they are comparing with. Jiang et al. (2026) use ~1k to 8k tokens per step for their language experiments, Lee at al. (2026) ~16k and Hu et al. (2025) ~64k. In contrast this paper frequently uses batches of 256k, 512k or 1M tokens. For a fixed token budget, these larger batches will result in significantly less gradient updates during PPT. I am concerned that the models may not have received enough gradient steps to acquire the underlying structre of the PPT data that previous work has found to be very beneficial for subsequent PT. Furthermore, the authors do no report PPT task performance and the end of PPT. It is difficult to distinguish whether PPT fails to give transferable benefits or if the setup is insufficient for the PPT data to be learnt sufficiently in the first place.
- Lee at al. (2026), Training Language via Neural Cellular Automata
- Hu et al. (2025), Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
Questions for authors
Figure 2:
- Given that Lee at al. (2026) explicitly state their method tunes learning rate, batch size and weight decay during PPT on NCA, could the authors investigate if retuning this hyperparameters when changing the initialisation restores the benefits of NCA. It seems particularly relevant given figure 5 shows NCA retaining 4.4% perplexity improvement under the spectral initialisation
- Could the authors investigate retaining the weights learned during PPT using Lee at al.'s original initialisation, while initialising the new embeddings etc at the start of PT using the alternative initialisation? This would clarify if the disappearance of NCA's advantage is really due to a better initialisation for PT or changes to what is acquired from training on NCA.
Batch sizes
- Please clarify if smaller batch sizes, closer to the prior work was explored during hyperparameter optimisation. If not what is the justification for making this change?
- Please clarify the number of gradient steps performed during PPT and the performance for the experiments. It is unclear whether the models receive enough optimisation steps to acquire useful computational structure.
5 comments
Summary
Dear Reviewer,
Just following up to summarize our response above:
- Confirmed our reproduction experiment is conducted in the way you reccomended it to be conducted in.
- Hyperparameter tuned the reproduction experiment - showed our claims hold.
- Provided additional compute matched experiments in the reproduction setting - strengthening the validity of the compute matched claims we make in the paper.
- Reducing the PPT batch size does not significantly improve PPT results.
- Report the PPT step counts are mostly within range of prior PPT work.
Thank you again for your review, we’d truly appreciate your thoughts on our responses, as we believe it strengthened the manuscript.
Thank you for the engagement, response and experiments.
These new results do strengthen the paper, however I remain concerned that the PPT setup has not been sufficiently optimised and searched through, including weight decay, duration etc. . It also remains unclear whether the models sufficiently learn the PPT tasks before transitioning to language pretraining.
I therefore maintain my original score.
Extension of Reproduction Experiment & Larger batch size justification
First of all, thank you for an engaging and thorough review. Here we address the points you raise:
Tuning hyperparameters of the reproduction experiment. The purpose of the NCA reproduction was to (1) demonstrate how sensitive PPT is to weight-initialization, a factor prior literature hasn’t explored (2) that such sensitivity extends beyond our own experimental settings (Figure 8), i.e. OpenWeb text, versus our FineWeb, and Llama versus our Nanochat. That being said, we have taken your suggestion and tuned the PPT and PT learning rates (lr) of this reproduction, each until the optimum is bracketed. We first tune the PT lr, then given this optimum, tune the PPT lr. Under the NCA’s optimal lr, at three seeds, we find the original 5.4% gain still largely collapses to ~0.7%, and hence tuning the hyperparameters for the NCA reproduction does not restore its 5.4% benefit.
Furthermore, we reproduce the C4-PPT control, both with the optimizer reset at the PPT-to-PT boundary, as in prior work ([1], [2], [3]), and with a unified optimizer across both phases, as in our work. The reset version is worse than the no-PPT baseline, as also observed in Figure 3 of [3], while the unified version matches NCA-PPT, further supporting the findings in our own setup. Note that only NCA's lr was tuned here, and we reuse it for the no-PPT and C4 runs. Despite these results, we agree that the wording "attributed to" was too strong, and have changed it to "sensitive to the weight initialization scheme" (line 17).
Initialization in the reproduction experiment. Thank you for raising this point. We want to clarify that we do run the experiment in the way you describe it. In Lee et al.’s NCA PPT setup, the token embeddings are initialized from a N(0,0.02) during the PPT phase, but at the transition to the PT phase, the token embedding table is instead initialized from a N(0,1). In our experiments we leave the PPT phase untouched, and change N(0,1) to N(0,0.02) at the transition. We now describe this with greater clarity in the Appendix B.
Larger batch sizes than prior PPT work. We too had observed that our hyperparameter search selected PPT batch sizes larger than those used in prior work. We double-checked this choice at the time and found no clear improvement from smaller batches. We now reconfirm this with additional experiments in the NCA 1% data-unconstrained setting, at the 1e18 and 3e18 scales. Here, we scale the PPT lr found at our 1M batch size to each smaller batch size with the square-root scaling rule [4], and run at three seeds per batch size.
As we can see, reducing the PPT batch size does not improve final validation BPB at either scale, and none of these settings brings NCA closer to the no-PPT baseline (0.9794 and 0.9234).
What are the PPT step counts? Below are the average PPT and PT steps per experiment. Prior work uses ~100 to 5,000 PPT steps (2M–164M tokens) ([1], [2], [3]). Most of our step counts fall within this range. The lower counts come from the data-constrained and 1% PPT experiments, which are different settings to those in prior work.
Why interleaving? The interleaving suggestion is made given the forgetting on the PPT distribution observed in Figure 4. We observe that what the PPT models learn about their own distribution is forgotten after the phase boundary. We have added more discussion on this in Section 8.
Again, thank you for your comments, we believe our manuscript has been improved because of your review.
[1] Hu et al. Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases. https://arxiv.org/abs/2502.19249
[2] Jiang et al. Procedural Pretraining: Warming Up Language Models with Abstract Data. https://arxiv.org/abs/2601.21725
[3] Lee et al. Training Language Models via Neural Cellular Automata. https://arxiv.org/abs/2603.10055
[4] Malladi et al. On the SDEs and Scaling Rules for Adaptive Gradient Algorithms. https://arxiv.org/abs/2205.10287