- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PRECOCIAL uses an LLM to rewrite symbolic data generators, scoring candidates through downstream learning while continuing a shared pretrained network. Accepted changes retain the network state; rejected changes roll it back. Matched-pretraining experiments assess the discovered programs on language, code, and planning tasks, alongside search-cost and transfer studies.
Critical strengths
- The fresh-network, fixed-60k-step comparisons in §4.1 separate final program quality from unequal pretraining accumulated during search. First-draft and random-mutation controls make the search contribution more interpretable.
- Planning tasks use exact checkers, and the paper reports both search GPU-hours and cross-domain transfer. These provide useful evidence beyond synthetic-data validation loss alone.
Critical weaknesses
- The tested models span only 0.15M–3.36M parameters. Appendix F reports shrinking fixed-budget gains with model size, with nonsignificant differences in that sweep. The practical reach of the proposed pretraining approach beyond these small models is therefore unresolved.
- Appendix A evaluates three discovered programs with three training seeds each, while Table 2 reports paired tests over the resulting runs. It is unclear whether inference accounts for dependence among runs sharing a searched program. This matters for confidence in improvements across independently discovered curricula.
Questions for authors
Please clarify the statistical unit in Tables 2 and 6. Do the main comparisons remain supported when existing runs are aggregated by search seed, or analyzed with uncertainty that accounts for the shared generator program?
2 comments
Thank you for your response. it addressed my immediate concern. However, taking into account the feedback from other reviews, I have decided to keep my current rating.
Thank you for the review. We answer both weaknesses and your question with new analyses and runs; the general response has the full tables.
Statistical unit (W2 and your question): the main comparisons hold with the program as the unit. You are right that the paper treated the 3 training runs of each searched program as independent. We re-ran the tests counting each program once (the mean of its 3 runs; exact sign-flip test across programs), and confirmed them with mixed models and cluster-robust errors:
Planning (Table 5) also holds: ahead of π₀ in 12/12 programs (p = .0005). The revised paper reports these program-level tests (Appendix B, Table 8) and uses them for the main-text comparisons. In every new scale experiment, seed s uses the program from search s, so each data point is a separate program. Our audit also found and fixed a driver bug affecting 4 of 78 searches; every claim holds without them (Table 2: 11/11 programs vs π₀, p = .001).
Scale (W1): the gain grows with model size. The submission stopped at 3.36M parameters because of compute limits. Since then we have trained models from scratch up to 1.02B parameters. The Sokoban program, searched once at 0.69M, beats the hand-written program at every size. Its lead grows from +6.1 points at 19.3M to +11.7 at 1.02B (14.0% vs 2.3%, 3/3 seeds, p = .004).