acceptodds
Under review as a conference paper at ICLR 2027

PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning

Abstract

Fully synthetic pretraining data generated by symbolic programs can bootstrap downstream training on naturalistic tasks, such as language modeling, but the gen- erating programs have so far been designed by hand. We introduce PRECOCIAL, a method for automatically optimizing programs that generate synthetic pretrain- ing data. Starting from an initial generator program, an LLM iteratively rewrites the program, and each rewrite is scored by pretraining a randomly-initialized transformer on its data and fine-tuning it on the downstream task. Because ev- ery score requires training, PRECOCIAL keeps a single network for the entire search, continuing its pretraining for each accepted program instead of training a new network per candidate, making it significantly more efficient. On four lan- guage and code datasets, at a fixed fine-tuning budget, searched programs reach lower perplexity than hand-designed curricula, published procedural generators, random program mutation and the LLM’s own first draft, and reach the same per- plexity as the hand-designed curricula with 9–33% fewer fine-tuning steps. We also observe that using a searched program generator from one domain transfers to datasets in other domains. Additionally, searched programs allow for better downstream performance on four different planning tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Pretraining, LLM, meta-learning, data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dPGVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRECOCIAL uses an LLM to rewrite symbolic data generators, scoring candidates through downstream learning while continuing a shared pretrained network. Accepted changes retain the network state; rejected changes roll it back. Matched-pretraining experiments assess the discovered programs on language, code, and planning tasks, alongside search-cost and transfer studies.

Critical strengths
  • The fresh-network, fixed-60k-step comparisons in §4.1 separate final program quality from unequal pretraining accumulated during search. First-draft and random-mutation controls make the search contribution more interpretable.
  • Planning tasks use exact checkers, and the paper reports both search GPU-hours and cross-domain transfer. These provide useful evidence beyond synthetic-data validation loss alone.
Critical weaknesses
  • The tested models span only 0.15M–3.36M parameters. Appendix F reports shrinking fixed-budget gains with model size, with nonsignificant differences in that sweep. The practical reach of the proposed pretraining approach beyond these small models is therefore unresolved.
  • Appendix A evaluates three discovered programs with three training seeds each, while Table 2 reports paired tests over the resulting runs. It is unclear whether inference accounts for dependence among runs sharing a searched program. This matters for confidence in improvements across independently discovered curricula.
Questions for authors

Please clarify the statistical unit in Tables 2 and 6. Do the main comparisons remain supported when existing runs are aggregated by search seed, or analyzed with uncertainty that accounts for the shared generator program?

2 comments

Thank you for the review. We answer both weaknesses and your question with new analyses and runs; the general response has the full tables.

Statistical unit (W2 and your question): the main comparisons hold with the program as the unit. You are right that the paper treated the 3 training runs of each searched program as independent. We re-ran the tests counting each program once (the mean of its 3 runs; exact sign-flip test across programs), and confirmed them with mixed models and cluster-robust errors:

Table 2: PRECOCIAL vsPrograms aheadSign-flip p
LLM's first draft12/12.0005
Random mutation11/12.0015
Procedural12/12.0005
Hand-written π₀12/12.0005
No pre-pretraining10/12.004

Planning (Table 5) also holds: ahead of π₀ in 12/12 programs (p = .0005). The revised paper reports these program-level tests (Appendix B, Table 8) and uses them for the main-text comparisons. In every new scale experiment, seed s uses the program from search s, so each data point is a separate program. Our audit also found and fixed a driver bug affecting 4 of 78 searches; every claim holds without them (Table 2: 11/11 programs vs π₀, p = .001).

Scale (W1): the gain grows with model size. The submission stopped at 3.36M parameters because of compute limits. Since then we have trained models from scratch up to 1.02B parameters. The Sokoban program, searched once at 0.69M, beats the hand-written program at every size. Its lead grows from +6.1 points at 19.3M to +11.7 at 1.02B (14.0% vs 2.3%, 3/3 seeds, p = .004).

  • A real gain, not a speed-up: the lead holds after 3× longer fine-tuning and grows with 10× more training data.
  • Transfer to an unseen task: used unchanged on mazes, which it never saw, the program beats every baseline in 3/3 seeds at every size from 19.3M to 1.02B.
  • The search matters: the searched program beats the same LLM's unsearched first draft by +6.3 to +9.4 points.
  • Pretrained models: on Qwen2.5-1.5B, the search raises BIG-Bench Hard from 66.6% (hand-written) to 79.6%, and planning from 13.3% to 22.0%.

Thank you for your response. it addressed my immediate concern. However, taking into account the feedback from other reviews, I have decided to keep my current rating.

Reviewer_QekfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRECOCIAL, which uses Claude Sonnet 4 to iteratively rewrite Python programs that generate synthetic pretraining data. Each candidate is scored by pretraining a small transformer on its data, fine-tuning it downstream, and measuring validation perplexity. To reduce cost, a single network is kept for the whole search: accepted programs continue its pretraining, and rejected ones are rolled back. On a 0.69M-parameter GPT-2, across 4 language/code datasets and 4 planning puzzles, searched programs beat hand-written curricula, procedural generators, random mutation and the LLM’s first draft under matched pretraining. They also transfer across datasets and partly across domains. The single-network protocol is reported to be both cheaper and better than training a fresh network per candidate.

Critical strengths
  • The experimental design is careful. Matched pretraining removes the confound of unequal pretraining steps. The first-draft and random-mutation baselines separate the search loop from the LLM’s prior. Results use paired tests, and null results are reported honestly. The task-hidden Blocksworld ablation is informative: a search that never sees the task does as well, suggesting the gains come from general structure rather than task knowledge.
  • The problem is worthwhile, automatically optimizing pre-pretraining data generators instead of hand-designing them. The single-network protocol is simple and applies to other program searches with expensive fitness.
Critical weaknesses
  • Scale and effect sizes limit significance. The models have 0.69M parameters and the downstream data is 10k documents. The gain over the LLM first draft is about 0.15 ppl on a base of about 5.3, and shrinks to 0.06–0.14 by 20k steps. Validation perplexity is still falling at 10k steps in 96% of fine-tunes, so the benefit is essentially pre-convergence speed-up. Table 15 shows the fixed-budget lead shrinking with model size, and none of those differences is significant with p between .076 and .351. PRECOCIAL ties no-pretraining on CodeParrot. The current evidence does not show relevance at realistic scale.
  • The algorithmic contribution is modest and the mechanism is unclear. The search strategy is statistically indistinguishable from FunSearch/AlphaEvolve, so the main novelty is the single-network protocol. Yet part of its “better” result comes, as the authors acknowledge, from more accumulated pretraining (up to 60k vs. 15k steps), and there is no control matched on total pretraining steps. Also, accepted candidates carry no more task-matching data than rejected ones, and their pretraining loss is higher. Whether the gains come from fitness-based selection, or simply from more diverse or novel data, is not disentangled.
Questions for authors
  • In 4.4, does the single network still win if each fresh network is trained to the same total number of pretraining steps?
1 comment

Thank you for the review. The general response has the full tables.

Your question (§4.4: does the single network still win if fresh networks get equal pretraining?) No, and we have revised the paper. Fresh networks with equal pretraining pick slightly better programs, but cost 2.1× the GPU time. Compared with the paper's fresh evaluator, the single network's programs are equally good (p = .56), as you suspected. The revised §4.3 (which absorbs the old §4.4) says the single network is the cheaper evaluator, not the better one.

This choice is secondary to the contribution. PRECOCIAL brings FunSearch/AlphaEvolve-style search to the programs that generate synthetic pretraining data, with how well a trained model then learns as the fitness. The closest work differs in what is searched or how it is scored: DataEvolve evolves curation strategies for natural web data, Montessori-Instruct trains a teacher LLM's weights to write instruction data, CADEvolve evolves generator programs but selects them by validity checks, and Eurekaverse evolves RL environments. We found no prior work that searches over data-generating programs scored by a model trained on their output. It is cheap: about 10 LLM proposals (≈$1) and 1.1 GPU-hours per search, which pays for itself on the first fine-tune at 1.02B.

Scale and effect sizes (W1). The submission stopped at 3.36M parameters because of compute limitations. We have now trained models up to 1.02B from scratch. One example is the Sokoban program searched with a 0.69M model beats the hand-written program at every size, and its lead grows from +6.1 to +11.7 points (3/3 seeds each). The gain is real, not a speed-up: it holds after 3× longer fine-tuning and grows with 10× more data. Strikingly, the same program improves mazes, a task it was never searched on (3/3 seeds ahead of every baseline at 19.3M, 85M, 303M and 1.02B).

Selection vs diversity (W2): the search is what matters.

  • LLM without selection. The same LLM's first draft stays near no pre-pretraining, while the searched program is ahead of it by +9.4, +6.3 and +8.0 points at 19.3M, 85M and 303M (3/3 seeds each).
  • Selection without the LLM. Random mutation falls short of the LLM search (11/12 programs, p = .0015).

Both the LLM's proposals and selection are needed.

Pretrained LLMs. With Qwen2.5-1.5B LoRA adapters as the evaluator, the search raises BIG-Bench Hard to 79.6%, from 21.0% for the untrained model, 66.6% for hand-written programs and 70.3% for the LLM's first draft (3/3 seeds). Every rewrite the searches kept adds step-by-step worked solutions (as in chain-of-thought), one of four kinds of change the writer's prompt suggests.

Reviewer_dm93OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRECOCIAL uses an LLM to search over programs that generate synthetic pretraining data. Each candidate is evaluated through synthetic pretraining and downstream fine-tuning. To reduce evaluation cost, the search continues pretraining a shared network and rolls back rejected updates. Experiments on language modeling and planning compare searched generators with handwritten curricula, an initial LLM draft, random mutations, and alternative search strategies. Matched-pretraining evaluations show improvements from the selected generators, with additional analyses of transfer, scaling, and cost.

Critical strengths
  1. Tables 2 and 5 evaluate the selected generators by training fresh networks with matched pretraining budgets. This addresses the concern that apparent gains simply come from accumulated training during search. Comparisons with the initial LLM draft and random mutations also help identify what iterative program modification contributes.
  2. The planning results give the approach support beyond perplexity. Under matched pretraining, the 8-puzzle solve rate rises from 0.172 with the handwritten generator to 0.246 with PRECOCIAL, while Sokoban rises from 0.019 to 0.086. These outcomes are checked through task execution and demonstrate useful downstream improvements.
  3. The Blocksworld experiment withholding task information from the proposing LLM is a useful control. Downstream scoring remains available, and the resulting solve rate is 0.196 versus 0.216 in the standard setting. Although this does not establish equivalence, it shows that useful search need not depend on giving the LLM task examples.
Critical weaknesses
  1. The paper does not establish when search reduces the total cost of reaching a downstream performance target. Table 19 reports 0.71 GPU-hours for PRECOCIAL versus 0.88 and 0.85 for fresh-network-per-candidate FunSearch and AlphaEvolve; with shared networks, the latter methods cost 0.62 and 0.67 hours in Table 4. These results support the cheaper evaluation protocol. They do not yet show when searching is preferable to using a fixed generator and spending the saved compute on training.
  2. Table 4 directly compares continued and fresh evaluators and finds lower search cost and better pooled downstream performance with continued evaluation. However, their final networks differ in training history and accumulated pretraining. The matched-pretraining results validate the selected generators, but do not establish how closely intermediate candidate rankings reflect generator quality at a common budget. I view this as an unresolved explanation of the search’s effectiveness, with less weight than the practical-cost concern.
  3. The evidence remains confined to small models. The main experiments use 0.69M parameters, and Appendix F extends the study to 3.36M, which is a useful addition. However, that sweep evaluates networks produced by the search loops rather than using the matched-pretraining protocol of the main tables. The gains also narrow with longer fine-tuning. The authors acknowledge this limitation, but it leaves the relevance to larger-scale pretraining uncertain.
Questions for authors
  1. Cost clarification: Including search, final pretraining, and fine-tuning, how many downstream uses are needed to reach a fixed performance target more cheaply than the handwritten-generator baseline?
  2. Additional analysis: How well do candidate rankings from the evolving evaluator agree with fresh-evaluator rankings at a common training budget?
  3. Additional experiment: Does a generator searched with the small model retain its advantage on a larger model under matched pretraining?
1 comment

Thank you for the analyses and questions. The general response has the full tables.

Q1 (cost). Break-even = search cost ÷ GPU-hours saved per fine-tune.

SettingUses to break even
303M, Sokoban (program searched at 0.69M)1st-2nd use
1.02B, Sokoban (program searched at 0.69M)1st use: within 30k steps, π₀ never reaches what PRECOCIAL reaches in 10k

We also gave π₀ one search's GPU time as extra pretraining (340k steps instead of 60k). This does not close the gap: PRECOCIAL stays ahead on language (12/12 pairs, p < .0001) and on the 8-puzzle, Sokoban and Countdown (9/9 pairs).

Q2 (rankings at a common budget). We re-scored the 197 candidates of the one-network vs fresh-network searches (Table 4 of the submission) from a common network state. On WikiText, the single network's ranking tracks ground truth as well as a fresh network's does (Spearman ρ = 0.53 vs 0.47), and the two rankings agree (ρ = 0.52). The programs the two evaluators select are equally good (retrained for 60k steps: 0.058 ppl apart, p = .56); the revised paper reports this retrained comparison (Table 20) rather than the rank analysis. We present the single network as the much cheaper evaluator, not the better one. Most importantly, the novelty of this paper is an LLM iteratively searching over programs that generate synthetic pretraining data, each scored by a model trained on its output; to our knowledge, no prior work does this.

Q3 (does the small-model program keep its advantage on larger models?) Yes; this is now our headline result. The Sokoban program searched once at 0.69M beats the hand-written program at 19.3M, 85M, 303M and 1.02B. Its lead grows from +6.1 to +11.7 points (14.0% vs 2.3% at 1.02B, 3/3 seeds, p = .004). The lead survives 3× longer fine-tuning and 10× more data. The program also transfers to mazes, a task it never saw, beating every baseline in 3/3 seeds at every size. At 303M, PRECOCIAL also beats every baseline on language (12/12 pairs vs π₀, p = .006).

Changes to the paper: a new Section 5 with these results, the break-even table (Table 21), the evaluator comparison (Table 20), a revised §4.3 (which absorbs the old §4.4), and an appendix on pretrained LLMs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers. The shared concern was scale. Because of limited compute, the submission only had models of up to 3.36M parameters. Since submitting, we have trained models more than 300× larger from scratch (up to 1.02B parameters) and applied PRECOCIAL to pretrained LLMs. PRECOCIAL holds up at scale, and on Sokoban its advantage grows with model size. All new runs: matched pretraining, each arm at its best learning rate, 3 seeds, rules fixed before each run.

1. A program searched with a tiny model helps models up to 1,500× larger. We searched once with a 0.69M model on Sokoban, then used that program unchanged to pretrain models from scratch. Test solve rate on the Sokoban family:

Pretrained from scratch on19.3M85M303M1.02B
No pre-pretraining7.2%7.7%7.4%9.0%
Hand-written program π₀10.0%7.8%8.9%2.3%
Procedural generators (Jiang et al.)11.4%11.2%9.4%2.1%
PRECOCIAL (searched at 0.69M)16.1%15.0%17.1%14.0%
Lead over π₀ (seeds ahead)+6.1 (3/3)+7.2 (3/3)+8.2 (3/3, p = .003)+11.7 (3/3, p = .004)

2. It transfers to problems it never saw. The Sokoban program never saw a maze, yet it beats every baseline on 10×10 mazes (Searchformer's second task):

Maze solve rate19.3M85M303M1.02B
Nothing1.3%1.1%1.7%1.5%
Hand-written π₀1.2%2.0%1.7%2.0%
Procedural generators1.3%1.4%3.1%1.3%
LLM's first draft (no search)0.9%2.0%1.8%0.9%
PRECOCIAL (Sokoban program)3.7%6.8%7.7%5.3%

3/3 seeds ahead of every baseline at every size. A program searched with a 0.69M model on one puzzle improves 1B-parameter models on another.

3. It is a real gain, not just faster learning. After 3× longer fine-tuning, the lead over π₀ stays: +8.5 points at 303M and +9.9 at 1.02B (p = .017). With 10× more training problems (303M), the lead grows from +8.0 to +12.2 points (28.6% vs 16.4%, p = .011).

4. The search matters, not just the LLM. The same LLM's first draft, used without search:

Sokoban family19.3M85M303M
LLM's first draft6.8%8.7%9.1%
PRECOCIAL16.1%15.0%17.1%
Gain from search (seeds)+9.4 (3/3)+6.3 (3/3)+8.0 (3/3)

At 19.3M they also beat five published generators (procedural, Dyck, ECA, LIME, UTM) on Sokoban and the 8-puzzle (3/3 seeds, p ≤ .04 in all ten).

5. Language holds at 303M. PRECOCIAL has lower test perplexity than π₀ in 12/12 dataset–seed pairs (p = .006), than no pre-pretraining in 12/12 (p = .001), and than the first draft in 10/12 (p = .0014).

6. Pretrained LLMs. Pretrained models do poorly on these tasks by themselves. In our test format, Qwen2.5-1.5B scores 21.0% on BIG-Bench Hard, 0.0% on constraint-solving puzzles and 1.4% on planning (29.7%, 4.2% and 2.0% with few-shot examples). We ran the same search, scoring each candidate with a fresh LoRA adapter on this model; final programs are tested on held-out test sets never used in the search:

Qwen2.5-1.5BNo PRECOCIALHand-written programsPRECOCIAL
BIG-Bench Hard (14 tasks)21.0%66.6%79.6% (3/3 seeds)
Constraint puzzles (10)0.0%38.7%48.8% (p = .004)
Planning (7 tasks)1.4%13.3%22.0% (p = .013)

On BIG-Bench Hard it also beats the LLM's first draft (70.3%, 3/3 seeds). Every rewrite the searches kept adds step-by-step worked solutions before the answer (as in chain-of-thought), one of four kinds of change the writer's prompt suggests. A BIG-Bench Hard program found with the 1.5B model also trains a 0.5B model better (62.9% vs 57.4% for hand-written and first draft, 3/3 seeds).

7. What the contribution is. PRECOCIAL applies the FunSearch/AlphaEvolve idea to a new object: the program that generates synthetic pretraining data, scored by how well a model trained on its data then learns. To our knowledge, no prior work searches for synthetic pretraining data this way, so we mapped the design space. (a) Search small, reuse large: a 0.69M search transfers to 1.02B and to an unseen task. (b) One continually trained network is the cheap evaluator: it picks programs as good as a fresh network per candidate, with 2.5× fewer pretraining steps and 17–20% less GPU time. (c) The same search improves pretrained LLMs. A search costs about 10 LLM proposals (≈$1) and 1.1 GPU-hours, far below published FunSearch/AlphaEvolve runs, and pays for itself on the first fine-tune at 1.02B.

The revised paper includes these results (Section 5; Appendices J–L).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.