- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines whether a short phase of pre-pretraining (PPT) on synthetic sequences continues to help language models at larger scales and longer pretraining horizons. It compares synthetic tasks, pretraining data mixtures, and model sizes, with a main grid of 500M–3B models and 21B pretraining tokens, selected 3B runs extended to 100B tokens, and a 7B Marin run reaching 75.5B tokens. Most settings show downstream improvements. Grammatical acceptability does not consistently track these gains, leading the authors to question the grammatical-prior account and propose long-range retrieval as an alternative explanation.
Critical strengths
1. The study addresses the important question of whether PPT benefits are confined to small models and short training runs. Gains remain at the extended endpoints: 3B models improve the downstream average by 1.6 points on C4 and 1.2 points on Marin at 100B pretraining tokens, while the 7B Marin model improves by 1.0 point at 75.5B tokens (Table 4). These results provide useful evidence for the practical relevance of PPT beyond earlier small-scale experiments.
2. The controls strengthen the main empirical finding. The natural-text Control uses the same 500-step preliminary phase, helping assess whether gains are explained merely by additional training. Bracket-based PPT outperforms this control in 10 of the 12 main configurations. The three seeds on Marin at 3B also yield downstream gains of approximately 1.9, 1.9, and 1.7 points (Appendix B.2). Comparisons across synthetic tasks and data mixtures further show that the content of the preliminary training matters, rather than all synthetic data being equally useful.
Critical weaknesses
1. Lower whole-text NLL does not by itself show better use of the earlier list. The paper averages NLL over the first list, the repeated list, and the remaining context (Appendix A.3). This average could fall because the first list or other text becomes easier to predict, even if prediction of the repeated list does not improve at all. The aggregate score therefore does not identify where the benefit occurs. Even an improvement on the repeated span should be checked for dependence on the correct earlier list to provide more direct support for the proposed retrieval explanation.
2. The mixed directions of the BLiMP subtask results need a more specific mechanistic interpretation. In the Marin 3B experiment in Table 12, Anaphor Agreement improves by 2.9, 7.9, and 4.7 percentage points across the three seeds, and Irregular Forms improves by 1.0, 4.0, and 5.9 points. Ellipsis instead declines by 2.3, 1.9, and 1.3 points, while the overall changes are only −0.3, +0.1, and +0.9 points. These results more directly show that PPT affects different linguistic phenomena differently, without a consistent overall improvement. The directionally consistent subtask changes do not by themselves establish a grammatical prior or statistical significance, but should be explicitly addressed in the “not as a grammatical prior” interpretation.
Questions for authors
- Does PPT’s advantage occur on the repeated list, and does it depend on information in the earlier list? Using the existing models, please report NLL changes relative to PT-Only separately for the first list, the repeated list, and the remaining text. If feasible, keep the scored repeated list fixed while replacing the earlier list with an unrelated list matched in length and format, and compare both models on the repeated span. This would more directly test whether PPT’s advantage depends on using the earlier information.
- How should the small overall BLiMP changes be reconciled with subtask gains in all three seeds? Table 12 shows positive changes for Anaphor Agreement and Irregular Forms in each seed, but negative changes for Ellipsis. Please explain whether these existing results suggest selective effects on different linguistic phenomena, and why this pattern supports the “not as a grammatical prior” interpretation. Would a more precise conclusion be that downstream gains are not accompanied by a consistent overall BLiMP improvement, even though some subtasks may improve?
Response to Reviewer omGa
Thank you for the insightful review and for two concrete, actionable suggestions. We address each point below and mark the corresponding changes in blue in the updated paper.
We originally reported whole-stimulus NLL to follow Hu et al. (2025), but agree that this average cannot show where the benefit occurs. As you suggest, we measure mean per-token NLL separately on the first list, the text before the first list, the text between the lists, and the repeated list, for all 12 scale-mixture pairs (Appendix B.3, Table 14). Accordingly, we observe the following:
Following your suggestion, we keep the scored list fixed and change only the list before it, using the control stimuli of Armeni et al. (2022) (Appendix A.3, Table 15). Each comparison pairs a stimulus in which the final list follows an unrelated list of the same length with one in which the same final list follows itself:
§5.2 now summarizes both analyses.
Yes, the results suggest selective effects, and we read this pattern as evidence against a general grammatical prior for three reasons (§5.2, Appendix B.2, Table 12):
Yes. Your suggested conclusion captures this pattern well, and we have adopted it in §5.2: "Downstream gains are therefore not accompanied by a consistent overall BLiMP improvement, even though some phenomena change consistently within a scale." (L386-388) The §5.2 takeaway (L405-407) now carries the same point.