acceptodds
Under review as a conference paper at ICLR 2027

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Abstract

Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned during PPT that transfers to natural language grammar. However, PPT has only been tested on models of at most 1B parameters and PT budgets below 2B tokens on predominantly web text. It is unknown whether PPT is effective at larger scales and under more realistic PT data mixtures that combine diverse sources (e.g., code and math). We therefore present a comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens. Our results demonstrate that the downstream performance and token efficiency gains of PPT persist at scale, e.g., saving at least 21B PT tokens at the 3B scale. However, in contrast to prior work, we find no consistent evidence that these gains stem from a grammatical prior. Downstream performance does not consistently align with grammatical acceptability across model sizes. Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval. Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent. Overall, PPT is a low-cost addition to PT, and future PPT task design should target long-range retrieval rather than natural language grammar.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
pre-pretraining, pre-training, language models, scaling, data mixtures, token efficiency, k-Shuffle Dyck

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_omGaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether a short phase of pre-pretraining (PPT) on synthetic sequences continues to help language models at larger scales and longer pretraining horizons. It compares synthetic tasks, pretraining data mixtures, and model sizes, with a main grid of 500M–3B models and 21B pretraining tokens, selected 3B runs extended to 100B tokens, and a 7B Marin run reaching 75.5B tokens. Most settings show downstream improvements. Grammatical acceptability does not consistently track these gains, leading the authors to question the grammatical-prior account and propose long-range retrieval as an alternative explanation.

Critical strengths

1. The study addresses the important question of whether PPT benefits are confined to small models and short training runs. Gains remain at the extended endpoints: 3B models improve the downstream average by 1.6 points on C4 and 1.2 points on Marin at 100B pretraining tokens, while the 7B Marin model improves by 1.0 point at 75.5B tokens (Table 4). These results provide useful evidence for the practical relevance of PPT beyond earlier small-scale experiments.

2. The controls strengthen the main empirical finding. The natural-text Control uses the same 500-step preliminary phase, helping assess whether gains are explained merely by additional training. Bracket-based PPT outperforms this control in 10 of the 12 main configurations. The three seeds on Marin at 3B also yield downstream gains of approximately 1.9, 1.9, and 1.7 points (Appendix B.2). Comparisons across synthetic tasks and data mixtures further show that the content of the preliminary training matters, rather than all synthetic data being equally useful.

Critical weaknesses

1. Lower whole-text NLL does not by itself show better use of the earlier list. The paper averages NLL over the first list, the repeated list, and the remaining context (Appendix A.3). This average could fall because the first list or other text becomes easier to predict, even if prediction of the repeated list does not improve at all. The aggregate score therefore does not identify where the benefit occurs. Even an improvement on the repeated span should be checked for dependence on the correct earlier list to provide more direct support for the proposed retrieval explanation.

2. The mixed directions of the BLiMP subtask results need a more specific mechanistic interpretation. In the Marin 3B experiment in Table 12, Anaphor Agreement improves by 2.9, 7.9, and 4.7 percentage points across the three seeds, and Irregular Forms improves by 1.0, 4.0, and 5.9 points. Ellipsis instead declines by 2.3, 1.9, and 1.3 points, while the overall changes are only −0.3, +0.1, and +0.9 points. These results more directly show that PPT affects different linguistic phenomena differently, without a consistent overall improvement. The directionally consistent subtask changes do not by themselves establish a grammatical prior or statistical significance, but should be explicitly addressed in the “not as a grammatical prior” interpretation.

Questions for authors
  1. Does PPT’s advantage occur on the repeated list, and does it depend on information in the earlier list? Using the existing models, please report NLL changes relative to PT-Only separately for the first list, the repeated list, and the remaining text. If feasible, keep the scored repeated list fixed while replacing the earlier list with an unrelated list matched in length and format, and compare both models on the repeated span. This would more directly test whether PPT’s advantage depends on using the earlier information.
  2. How should the small overall BLiMP changes be reconciled with subtask gains in all three seeds? Table 12 shows positive changes for Anaphor Agreement and Irregular Forms in each seed, but negative changes for Ellipsis. Please explain whether these existing results suggest selective effects on different linguistic phenomena, and why this pattern supports the “not as a grammatical prior” interpretation. Would a more precise conclusion be that downstream gains are not accompanied by a consistent overall BLiMP improvement, even though some subtasks may improve?
1 comment
AuthorsOpenReview ↗

Response to Reviewer omGa

Thank you for the insightful review and for two concrete, actionable suggestions. We address each point below and mark the corresponding changes in blue in the updated paper.


Does PPT's advantage occur on the repeated list, and does it depend on information in the earlier list? Using the existing models, please report NLL changes relative to PT-Only separately for the first list, the repeated list, and the remaining text.

We originally reported whole-stimulus NLL to follow Hu et al. (2025), but agree that this average cannot show where the benefit occurs. As you suggest, we measure mean per-token NLL separately on the first list, the text before the first list, the text between the lists, and the repeated list, for all 12 scale-mixture pairs (Appendix B.3, Table 14). Accordingly, we observe the following:

  • The PPT advantage occurs mainly on the repeated list at 500M and 1B. k-Shuffle Dyck lowers repeated-list NLL in 11 of 12 pairs, stably in 8. At 500M and 1B, the repeated-list reduction exceeds the first-list change in all eight pairs, and NLL between the lists falls by at most 0.10. At 3B, this holds on only one of four mixtures, and spans that cannot be copied also improve (e.g., −0.30 before the first list on C4). The retrieval-specific gain is therefore clear at 500M and 1B and weaker at 3B.
  • It comes mainly from the PPT data, not from the extra optimization steps. The in-domain text control, which runs the same 500 warm-up steps on held-out PT text, lowers repeated-list NLL less than k-Shuffle Dyck in 10 of 12 pairs, with 5 stable reductions against 8 (none at 500M). As with downstream performance (§5.1), the repeated-list advantage is therefore mainly attributable to the synthetic PPT data.

If feasible, keep the scored repeated list fixed while replacing the earlier list with an unrelated list matched in length and format, and compare both models on the repeated span.

Following your suggestion, we keep the scored list fixed and change only the list before it, using the control stimuli of Armeni et al. (2022) (Appendix A.3, Table 15). Each comparison pairs a stimulus in which the final list follows an unrelated list of the same length with one in which the same final list follows itself:

  • The PPT advantage depends on the earlier list. With an unrelated earlier list, the advantage of k-Shuffle Dyck over PT-Only shrinks from up to 0.34 to at most 0.11. The advantage is larger with a matching earlier list in 11 of 12 pairs, most clearly at 500M and 1B, and weaker at 3B, where the repeated-list advantage is also smallest. For the in-domain text control, this difference is smaller than for k-Shuffle Dyck in 11 of 12 pairs. PPT therefore gains more than PT-Only from a matching earlier list, showing that its advantage depends on information earlier in the context.

§5.2 now summarizes both analyses.


Please explain whether these existing results suggest selective effects on different linguistic phenomena, and why this pattern supports the "not as a grammatical prior" interpretation.

Yes, the results suggest selective effects, and we read this pattern as evidence against a general grammatical prior for three reasons (§5.2, Appendix B.2, Table 12):

  1. The effects go in both directions. At 3B, 8 of 12 phenomena keep their sign across all three seeds, against about 3 expected by chance. Four consistently improve (anaphor agreement, irregular forms, control/raising, and island effects), and four consistently decline (determiner-noun agreement, argument structure, ellipsis, and filler-gap). PPT therefore does not shift grammatical acceptability in one direction.
  2. They are not stable across scales. With three seeds at 500M and 1B on Marin as well, only 4 and 5 of 12 phenomena keep their sign, close to chance, and none of the four phenomena that improve at 3B does so consistently at 500M or 1B. For example, anaphor agreement changes by +2.1, −2.9, and −2.6 across the three 1B seeds. Only determiner–noun agreement keeps its sign at every scale, and it declines. The selective effects are therefore not a stable property of PPT.
  3. There is no net gain. The BLiMP overall change is −0.3, +0.1, and +0.9 across our 3B seeds, against a downstream gain of +1.7 to +1.9 in every seed (Tables 11 and 12). The consistent phenomena therefore do not add up to an overall improvement.

Would a more precise conclusion be that downstream gains are not accompanied by a consistent overall BLiMP improvement, even though some subtasks may improve?

Yes. Your suggested conclusion captures this pattern well, and we have adopted it in §5.2: "Downstream gains are therefore not accompanied by a consistent overall BLiMP improvement, even though some phenomena change consistently within a scale." (L386-388) The §5.2 takeaway (L405-407) now carries the same point.

Reviewer_Fyk2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits synthetic pre-pretraining (PPT), asking whether its benefits persist beyond the small models, short pretraining runs, and predominantly web-text corpora used in earlier work. The study spans models from 500M to 7B parameters, four pretraining mixtures, and several PPT tasks, with selected 3B runs extended to 100B tokens. It finds that a short synthetic warm-up can yield downstream improvements after substantial subsequent pretraining. It also questions the grammatical-prior explanation for PPT and proposes long-range retrieval as an alternative explanation.

Critical strengths
  1. The scale extension answers a useful empirical question.

PPT itself is an existing approach; the contribution here is evidence that its benefits can survive substantially larger models and longer training. In Table 4, the 3B C4 and Marin runs retain gains of 1.6 and 1.2 points at 100B tokens, respectively. On Marin, the PPT model scores 62.3 after 63B pretraining tokens, versus 60.8 for the baseline at the same budget and 62.0 for the baseline at 84B tokens (Table 15). This checkpoint comparison suggests a meaningful opportunity for token savings. The results make a stronger case for PPT as a potentially useful training ingredient than earlier short-run experiments could provide.

  1. The controls help establish that the result is more than extra training or a peculiarity of one corpus.

The natural-text warm-up control uses the same preliminary training duration and generally yields smaller gains than synthetic PPT. The three-seed 3B Marin experiment also supports the downstream improvement beyond a single initialization. The mixture experiments address a relevant concern: adding code and mathematics might make synthetic structural training redundant. Raising Marin's mathematics share from 1.3% to 17% leaves the PPT gain approximately unchanged (Table 3a), showing that a substantial increase in mathematics does not necessarily remove its value.

Critical weaknesses
  1. The retrieval explanation is stated more strongly than the evidence supports.

Appendix A.3 specifies that the retrieval metric averages NLL over the entire stimulus, rather than only the repeated span. A lower value can therefore reflect changes in general language modeling as well as retrieval. The PPT tasks also differ along several dimensions, so comparing their transfer does not isolate retrieval demand. There are relevant exceptions: the 100B C4 run improves downstream performance despite slightly worse retrieval NLL (Table 4), and one of the three Marin seeds shows the same qualitative discrepancy (Table 13). These observations support retrieval as a plausible hypothesis, but do not establish it as the necessary mechanism behind the gains. Similarly, inconsistent BLiMP improvements weaken the tested grammatical-prior account without excluding structural inductive biases more broadly. This matters because the mechanism is a prominent part of the paper's claimed contribution and motivates its task-design recommendation.

  1. The mixture conclusions overstate how well the conditions for success are understood.

The experiments establish that PPT can remain useful with code and mathematics present. They do not establish that the presence of web text is sufficient, or that gains disappear only when web text is removed. OLMo3 contains substantial web text yet shows a small negative gain at 1B and essentially no gain at 3B. Section 6 explicitly leaves this exception unexplained. The web-removal experiment is informative, but the general statements in the abstract and conclusion should acknowledge this unresolved dependence on the corpus. This limits the practical guidance: the paper shows several settings where PPT helps, while offering less evidence about whether it will help a new training mixture. Most configurations have only one training seed, and the 7B extension covers one mixture and one PPT task, which further limits the breadth of that guidance.

Questions for authors

OLMo3 contains substantial web text, yet PPT provides little or no benefit at 1B and 3B. How is this consistent with the claim that gains diminish only when web text is absent? Should the conclusion explicitly acknowledge this exception and the unresolved dependence on the pretraining corpus?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Fyk2 (Part 1)

Thank you for the careful review and for recognizing the value of the scale extension and the controls. We address each point below and mark the corresponding changes in blue in the updated paper.


These observations support retrieval as a plausible hypothesis, but do not establish it as the necessary mechanism behind the gains.

We agree that our evidence supports long-range retrieval as a candidate mechanism, not as the necessary one, and we update the claim throughout the manuscript (see the end of this post).

Appendix A.3 specifies that the retrieval metric averages NLL over the entire stimulus, rather than only the repeated span. A lower value can therefore reflect changes in general language modeling as well as retrieval.

We followed Hu et al. (2025) for comparability, but agree that this average mixes retrieval with general language modeling. Each stimulus of Armeni et al. (2022) contains a noun list, intervening text, and the same list again. We therefore measure NLL separately on each span (Appendix B.3, Table 14). The first list cannot be copied from context, whereas the repeated list can. Accordingly, we observe the following:

  • k-Shuffle Dyck lowers repeated-list NLL in 11 of 12 scale-mixture pairs (8 stable, by up to 0.34). The repeated-list reduction exceeds the first-list change in all eight pairs at 500M and 1B, but in only one of four at 3B, where spans that cannot be copied also improve. At 500M and 1B, the gain therefore mainly reflects in-context retrieval rather than general language modeling; at 3B, this evidence is weaker.
  • The in-domain text control lowers repeated-list NLL less than k-Shuffle Dyck in 10 of 12 pairs, with 5 stable reductions against 8. The gain therefore comes mainly from the synthetic PPT data rather than from the extra training steps.
  • When the same list follows an unrelated first list, the advantage of k-Shuffle Dyck shrinks from up to 0.34 to at most 0.11 (Table 15). The gain therefore depends on information earlier in the context.

There are relevant exceptions: the 100B C4 run improves downstream performance despite slightly worse retrieval NLL (Table 4), and one of the three Marin seeds shows the same qualitative discrepancy (Table 13).

We examine both runs in Appendix B.3 ("Runs with Worse Whole-Stimulus NLL"). For the 3B C4 run at 100B tokens, whole-stimulus NLL is slightly worse (+0.008), but repeated-list NLL improves (−0.144). For the three 3B Marin seeds, repeated-list NLL improves in all three (−0.054, −0.019, and −0.086), including the seed with worse whole-stimulus NLL, although its reduction is not stable. In both cases, the worse whole-stimulus NLL does not come from the repeated list, the span that measures retrieval.

The PPT tasks also differ along several dimensions, so comparing their transfer does not isolate retrieval demand.

We agree that our PPT tasks differ in several respects and that comparing them cannot isolate retrieval demand. The updated §6: L486-491 states this, names one such difference (Set is the only task in which a separator splits each sequence into an input and an output), and leaves a controlled test of retrieval demand to future work. Within these limits, the comparison points toward retrieval. The PPT tasks other than Set lower the ratio of repeated- to first-list NLL in 14 of 17 task-mixture pairs, whereas Set, the only harmful task, raises it on all four mixtures (Appendix Table 16).

Similarly, inconsistent BLiMP improvements weaken the tested grammatical-prior account without excluding structural inductive biases more broadly.

We agree that inconsistent BLiMP gains do not rule out structural inductive biases in general. Our target is the grammatical prior of Hu et al. (2025), which they evidence with BLiMP. The updated §5.2 explicitly states this scope (footnote 5) and adds structural probes as a complementary test (Appendix B.4).


To state retrieval as a candidate mechanism, we have updated the manuscript as follows:

  • Abstract: "Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval." → "Instead, downstream gains align better with long-range retrieval than with grammatical acceptability, suggesting retrieval as a candidate mechanism." (L22-24) The final sentence (L26-27) now reads "future PPT tasks may benefit more from targeting long-range retrieval than natural language grammar."
  • Contributions (§1): "Instead, we find that downstream gains arise only from PPT tasks that improve long-range retrieval." → "Instead, downstream gains align better with long-range retrieval than with grammatical acceptability, and the retrieval gain depends on information earlier in the context." (L85-87)
  • Conclusion: "We show instead that PPT is effective when the PPT task requires [...]" → "Our results instead suggest that PPT is effective when the PPT task requires retrieving tokens from earlier positions in a sequence." (L536-537)
AuthorsOpenReview ↗

Response to Reviewer Fyk2 (Part 2)

OLMo3 contains substantial web text, yet PPT provides little or no benefit at 1B and 3B. How is this consistent with the claim that gains diminish only when web text is absent? Should the conclusion explicitly acknowledge this exception and the unresolved dependence on the pretraining corpus?

Thank you for this question. We agree that the general statement in the abstract and conclusion should account for OLMo3. Prompted by your question, we have identified what distinguishes the OLMo3 sample from the other mixtures and updated the statement accordingly (§6, Appendix C.1).

Since neither the math share nor the web share explains the gap, we inspect the OLMo3 training data itself. This reveals heavy document repetition that none of the other mixtures shares. About 63% of the OLMo3 tokens we train on are exact repeats, and a 21B-token run therefore sees fewer than 8B unique tokens. The web components of the other mixtures contain at most 4% exact repeats. The repetition is present in the released 150B-token sample of the Dolma 3 Mix (allenai/dolma3_mix-150B-1025) and is not introduced by our pipeline.

We then train PT-Only and k-Shuffle Dyck at 3B on OLMo3 with the duplicates removed, keeping only the first occurrence of each document ID and all other settings unchanged (Table 22). After this step, only 0.04% of tokens remain exact repeats. PPT now improves the downstream average by +1.6 (SD 0.3; positive at all six checkpoints), against +0.0 on the original OLMo3 and +1.3 to +1.9 on C4, SmolLM3, and Marin. As on the other mixtures, the largest gain is in language modeling (+4.1). Deduplication also raises the PT-Only average itself from 42.8 to 49.5. The OLMo3 exception is therefore specific to the released sample and disappears once its duplicates are removed.

We also agree that our four mixtures cannot establish web text as sufficient for PPT to help on an untested mixture. We therefore restrict the claim to deduplicated PT data, and the OLMo3 result shows that properties beyond composition can matter. We have updated the manuscript as follows:

  • Abstract: "Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent." → "Finally, on deduplicated PT data, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent." (L24-26)
  • Contributions (§1): we add "The gain also vanishes on a PT mixture with heavy document repetition and returns after deduplication." (L90-91)
  • Conclusion: "Finally, we demonstrate that PPT gains are insensitive to the share of code and math in the PT corpus, and diminish only when web text is absent." → "Finally, we demonstrate that on deduplicated PT data, PPT gains are insensitive to the share of code and math in the PT corpus, and diminish only when web text is absent." (L537-539)
  • §6: "We leave a direct ablation of OLMo3, and the property responsible for this gap, for future work" is replaced by the repetition audit and the deduplicated run results (L455-460).

Most configurations have only one training seed, and the 7B extension covers one mixture and one PPT task, which further limits the breadth of that guidance.

We extend the three-seed analysis to 500M and 1B on Marin, where PPT improves the downstream average in all nine seed pairs (Appendix B.2). The 7B extension remains limited to one mixture and one PPT task because of compute constraints (footnote 4).

Thank you for the additional experiments. The span-level retrieval analyses and the deduplicated OLMo3 experiment substantially address my main concerns. Retrieval remains a candidate mechanism, but I appreciate the more cautious framing. The new evidence strengthens the empirical contribution, and I will raise my rating from 2 to 3.

Thank you for reading our responses and for raising your rating. Your comments on the retrieval metric/explanation and the OLMo3 exception resulted in conducting the span-level analysis and the OLMo3 deduplication finding. These have substantially strengthened the paper. Thank you!

Reviewer_Vu3DOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper tests whether LMs still benefit from the pre-pretraining when both the training data and model sizes are scaled up. They manipulate the training data mixture, pre-pretraining languages, and model sizes and evaluated the models 4 downstream tasks. They find that pre-pretraining still survive scaling up to 7B models and 100B trainign data. They then test the hypothesis that prepretraining is useful because it adds a grammatical prior to LMs by evaluating LMs on BLiMP and they find that negative results.

Critical strengths
  • This paper addresses an important question about pre-pretraining and supports its conclusions with robust and extensive evidence.

  • The paper carefully and systematically controls for various engineering choices. Specifically, the manipulations of the pretraining data mixture, the selection of pre-pretraining languages, and model size are important for addressing the question. The experiment design is very clean.

  • Each experiment has clear takeaways, and the paper is generally very easy to follow.

Critical weaknesses

My main concern is with Section 5.2. Hu et al. (2025) also show that improvements on BLiMP following pre-pretraining are not very obvious (Figure 2 has the error bars overlapping with each other). So the inconsistent improvements found in this paper is not very surprising to me. And I'm worried BLiMP performance alone may not provide sufficient evidence for the model’s grammatical prior hypothesis. Assessing grammatical priors may require complementary analyses, such as structural probing (e.g., https://aclanthology.org/N19-1419.pdf). However, I do not consider this a critical issue for the paper. A discussion of the relationship between BLiMP performance and linguistic priors would likely be sufficient.

Questions for authors

I think the authors should also clarify what 1 point of improvement really means for pretraining. Most of the improvements of across all settings are minimal, though positive. Does that mean that pre-pretraining is worth the efforts in scaling? It would be nice if the authors can provide a discussion on this in the updated version.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Vu3D

Thank you for the positive assessment and constructive suggestions. We address each point below and mark the corresponding changes in blue in the updated paper.


I'm worried BLiMP performance alone may not provide sufficient evidence for the model's grammatical prior hypothesis.

We agree that BLiMP alone cannot rule out every grammatical prior. We use it because Hu et al. (2025) present BLiMP gains as their evidence for the prior (§4.2: L296-L300). The updated manuscript states this scope explicitly (footnote 5) and ends the BLiMP analysis with a narrower claim (§5.2: L388-389): "At our tested scales, the behavioral evidence of Hu et al. (2025) for a grammatical prior fails to replicate."


Hu et al. (2025) also show that improvements on BLiMP following pre-pretraining are not very obvious [...]

We agree that the BLiMP differences are small, as the overlapping error bars in their Figure 2 suggest. We respectfully note, however, that Hu et al. (2025) interpret these differences as gains: they report that pre-pretraining on all four formal languages “improves accuracy in grammaticality,” whereas pre-pretraining on natural language does not, and they attribute this to representations useful for modeling hierarchical structure (§3). This interpretation is what our paper tests. To check that the gap between BLiMP and downstream gains does not come from seed noise or from our choice of PPT tasks, we add two analyses.

  1. Three seeds at 500M, 1B, and 3B (Appendix B.2, Tables 11 and 12).
    We train two more seeds of PT-Only and k-Shuffle Dyck on Marin at 500M and 1B, matching the three seeds at 3B. PPT improves the downstream average in all nine seed pairs (+0.9 ± 0.4, +1.4 ± 0.5, and +1.9 ± 0.1), whereas the BLiMP change is smaller than its SD across seeds at every scale (+1.0 ± 1.3, +0.4 ± 1.2, and +0.2 ± 0.6). The gap between BLiMP and downstream gains is therefore not due to seed noise.
  2. More PPT tasks (§6, Table 3b, Appendix C.2).
    We add symbolic music (Papadimitriou & Jurafsky, 2020) and a mixture of k-Shuffle Dyck, MP-Struct Core, and NCA at 3B. Across the 13 task-mixture pairs on C4, SmolLM3, and Marin for the tasks that help, BLiMP changes do not rise with downstream changes (r = −0.57). The gap is therefore not specific to k-Shuffle Dyck.

Both results support our main claim: PPT's downstream gains are not accompanied by overall BLiMP gains when scaling.


Assessing grammatical priors may require complementary analyses, such as structural probing [...]. A discussion of the relationship between BLiMP performance and linguistic priors would likely be sufficient.

Thanks for the helpful suggestion. The updated §5.2 discusses this relationship. It states which prior BLiMP can test (footnote 5) and adds structural probes (Hewitt & Manning, 2019), as you suggest (§5.2: L383-386, Appendix B.4, Tables 17 & 18). PPT improves most probe metrics at 500M, but this gain does not persist at 3B, where results are mixed across metrics and depend on the probing protocol, despite an average downstream gain of 1.3 points. As probes measure what is linearly decodable rather than what the model uses (Belinkov, 2022), we treat this result as support for the BLiMP result rather than as proof on its own (Appendix B.4).

(Belinkov, 2022): Yonatan Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 48(1):207–219.


I think the authors should also clarify what 1 point of improvement really means for pretraining. [...] Does that mean that pre-pretraining is worth the efforts in scaling?

We appreciate this question. The updated manuscript adds a paragraph on the practical value of PPT before the Conclusion (§6, "Practical Value of PPT"), based on four points:

  1. Cost. PPT uses about 1B tokens, 5% of a 21B-token run and about 1% of a 100B-token run.
  2. Token savings. On Marin at 3B, PPT after 63B PT tokens exceeds PT-Only after 84B on the downstream average (62.3 vs. 62.0), saving at least 21B tokens (25% of the PT-Only budget), or about 1,136 A100 GPU hours against 56.5 hours for PPT itself (Appendix Table 20, Appendix A.2).
  3. Gain at an equal budget. At 100B tokens on Marin at 3B, PPT improves the downstream average by +1.2 (Table 4).
  4. Comparison with model scaling. Tripling model size from 1B to 3B raises the PT-Only average on Marin by 3.25 points (three-seed means). PPT adds 1.85 at 3B with no extra parameters, more than half of that gain.

Therefore, for about 1% of the PT compute, PPT either reaches the same performance with at least 25% fewer PT tokens or yields a better model at the same budget.

It's interesting to see the structural representation becomes less stable at scale. All my concerns have been addressed. Thank you! I'll keep the score.

Thank you for your prompt response and for your continued support of the paper. We are glad that the response and updates to the paper have addressed your concerns. Your suggestion to add structural probes made the analysis more complete and interesting!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the reviewers

We thank the reviewers for their constructive feedback. We are glad that all three rate the presentation as "clear and easy to follow" (Vu3D, Fyk2, omGa), and two of three report gaining "a substantially deeper understanding of the problem or question" (Vu3D, omGa). The reviewers find the question important or useful (Vu3D, Fyk2, omGa) and value our controls (Vu3D, Fyk2, omGa), with Vu3D describing the design as "very clean". On soundness, Vu3D has "no substantive concerns", and Fyk2 and omGa raise "minor or fixable concerns", which we address below. The main concerns are that (1) our explanatory claims are stated strongly and that (2) the OLMo3 exception needs an explanation. We summarize below how we have addressed each of them in the updated manuscript (changes in blue).

1. BLiMP alone may not suffice to reject a grammatical prior [Vu3D Weakness; Fyk2 Weakness 1; omGa Question 2]

  • We explicitly scope the claim to the prior that Hu et al. (2025) evidence with BLiMP (footnote 5), and adopt omGa's formulation in §5.2 (L386-388 & L405-406)
  • We add three seeds at 500M and 1B in addition to 3B on Marin (Appendix B.2), which also addresses the single-seed concern of Fyk2, showing that the gap between downstream and BLiMP performances is not seed noise. PPT improves the downstream average in all nine seed pairs (+0.9, +1.4, and +1.9), whereas the mean BLiMP change stays below its SD across seeds at every scale.
  • We add two more PPT tasks at 3B, symbolic music (+1.0) and a mixture of three tasks (+1.5 on Marin), showing that the gap is not specific to k-Shuffle Dyck (§6: L464-494, Appendix C.2).
  • Following Reviewer Vu3D's suggestion, we add structural probes, testing model representations directly (Appendix B.4). Probe gains at 500M do not persist at 3B, where results are mixed and depend on the probing protocol, despite the downstream gain.

2. Whole-stimulus NLL does not necessarily isolate retrieval [Fyk2 Weakness 1; omGa Question 1]

  • Splitting NLL by span (Appendix B.3), PPT lowers repeated-list NLL in 11 of 12 scale-mixture pairs (8 stable), with the other spans changing much less at 500M and 1B; at 3B, spans that cannot be copied also improve.
  • When the same list follows an unrelated first list, the advantage largely disappears (Table 15). The gain therefore depends on information earlier in the context, not general language modeling.
  • We now present retrieval as a candidate mechanism rather than an established one (Abstract: L22-24, §1: L85-87, §6: L480-493, and Conclusion: L536-537).

3. OLMo3 contains web text yet shows no PPT gain [Fyk2 Weakness 2]

  • We show that the released OLMo3 mixture we train on is heavily duplicated (about 63% of tokens are exact repeats, against at most 4% in the other mixtures; Appendix C.1).
  • With duplicates removed, PPT improves the 3B OLMo3 average by +1.6 (SD 0.3), against +0.0 before (§6: L455-460, Appendix C.1, Table 22).
  • The data-mixture claim in the Abstract, Introduction, and Conclusion is now restricted to deduplicated PT data (L24, L90-91, & L537).

4. The practical meaning of a ~1-point gain [Vu3D Question]

  • We add a new paragraph in §6 ("Practical Value of PPT"), showing that, for about 1% of the PT compute, PPT saves at least 21B PT tokens (~1,136 A100 GPU hours) on Marin at 3B, or improves the average by +1.2 at the same 100B-token budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.