- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a novel (to the best of my knowledge) pretraining strategy that uses no natural data. The training sequences are synthetic byte sequences created by programs sampled from a generator. An RL-based scheme pushes the generator to create programs that are structured and aligned with the learner's trajectory. The main result is that the resulting model achieves nontrivial performance across a wide range of datasets, including images, text, and code. Furthermore, performance scales with compute as in Neural Scaling Laws, suggesting the possibility of scaling up this procedure to larger data and model scales.
Critical strengths
-
This work contributes significantly to the line showing that nontrivial predictive performance can emerge without natural-data pretraining. The introducion of an adaptive self-play strategy improves on fixed program sampling and seem to transfer across several modalities. I find this a compelling contribution to understanding how synthetic training data can support broadly useful learning.
-
I found the paper particularly enjoyable to read. The central research question is clear and well motivated. The text provides useful intuition for the proposed approach. This clarity makes both the method and the significance of the empirical findings accessible, strengthening the paper’s value to a broad ICLR audience.
Critical weaknesses
I did not find any critical weakness, although the authors should have tried to connect their results to ideas from the theoretical In-Context learning literature (see question below). I also added some minor concerns about the presentation in the questions section below.
Questions for authors
I believe that the submission as it stands is worthy of acceptance, but the extreme coarseness of the review form does not allow me to give a clearly positive assessment that could still be improved through feedback. In particular, the instruction to ask only assessment-changing questions is too restrictive to allow for general feedback that could improve the paper and contribute to a useful scientific discussion, without changing my assessment. This is a shame as it obstructs thoughtful reviewing and meaningful author-reviewer discussions aimed at improving the paper. I hope it will not become standard practice in future ML conferences.
Nevertheless, since I spent some time going through the paper, I will post my feedback here. If the authors reply, we will hopefully make something productive of this nonsensical review process.
Main questions
Comparison with ICL
According to some, the in-context learning phenomenon where a model infers a task from examples in the prompt is only an instance of a more generic form of out-of-distribution generalisation (see here for the main idea, here for a broad overview). In simple terms, when a learner is trained on examples drawn from a distribution over tasks, the learner might learn a generic predictive strategy instead of specialising to the individual tasks presented during training. This scenario is similar to the setting of this paper, as (modulo replay) every sentence is drawn from a new program. I wonder if it's possible to interpret the main observation of this paper as the model having learned some inference strategy over programs taken from a certain distribution. For example, even a model that infers the best Markov-process approximation of language would have nontrivial performance on real text. I understand that, in this case, the distribution over programs is evolving with training, but I would be curious to hear the author's idea on this connection.
Self-play with PCFG
The authors use a Brainf*ck-like language to generate programs, so as to guarantee, via Turing completeness, the representability of any data-generating process. However, the data on PCFG pretraining suggests that there might be a preference towards PCFG-like data-generating processes for text and code. Have the authors thought of changing the architecture of the program generator to incorporate a preference towards particular classes of data-generating processes?
Clarification requests and other minor concerns
-
The sketch explaining the generator reward in Figure 1 is far less clear than anything else in the first 5 pages or so. The in-text explanation before equation (2) is much simpler, so I hope the authors can find a better illustration.
-
I am not an expert in policy-gradient RL. I would have liked to have some intuitive description of the choices listed on page 5, between line 226 and equation (5). There was a similar explanation about the particular choice of reward on the previous page, which worked quite well. Even something at that level would be great to have.
-
In the "explaining scaling laws" section, I think that the distinction between and should be commented on immediately after equation (7). This is, I believe, the most important point of this section, and also a subtle one, which the sentence immediately after equation (7) does not clarify. Specifically, I agree with the author's intuition that scaling up should increase both and , but this cannot be true in general: think, for instance, of data memorisation, or of the aforementioned ICL phenomenon where learners transition from generic predictive strategies to specialising to the individual tasks seen in training. Without a more specific definition of and and/or a more detailed comment on when they are expected to increase with , the paragraph "comparing self-play and natural-data exponents" could be misleading.
-
The very last sentence of the main "future work. We defer a broader discussion to Appendix." does not seem to make much sense.
3 comments
Q3.3: Universal and contingent data in the scaling-law ansatz
Without a more specific definition of and and/or a more detailed comment on when they are expected to increase with , the paragraph "comparing self-play and natural-data exponents" could be misleading.
We appreciate this thoughtful observation. We agree that the distinction between universal and contingent information is subtle, and that their growth with dataset size should be presented as an assumption rather than a general guarantee.
Our intended interpretation of and is as effective information resources accessible to a learning algorithm: contingent information specific to the target distribution and universal predictive structure shared across distributions. We recognize that defining these quantities independently of the learning procedure, without sacrificing the generality of the framework, is challenging.
The assumption that both quantities increase with is most natural when additional data consists of fresh samples from a fixed underlying distribution, rather than repeated exposure to a finite corpus. As the reviewer points out, memorization or a transition toward task-specific strategies can violate this intuition. We agree that these distinctions deserve explicit discussion.
We will therefore make three clarifications in the revision:
- Clarify the interpretation of and immediately after Equation 7, emphasizing that they represent effective information available to the learner rather than simply raw dataset sizes.
- Explicitly identify Equations 9 and 11 as assumptions. In particular, Equation 9 assumes that self-play continues to discover additional universal structure as training progresses. This is an empirical hypothesis supported by our observations, not a theoretical guarantee.
- Qualify the comparison between self-play and natural-data scaling exponents to clarify the conditions under which the proposed decomposition is expected to apply.
We thank the reviewer for identifying this important subtlety. We believe these changes will make the interpretation of our scaling results more precise.
Q3.4: Final sentence of the discussion
The very last sentence [...] does not seem to make much sense.
Thank you for catching this editing artifact! We will correct the sentence and use the additional space to expand the broader discussion, including the connection to theoretical ICL raised above.
Thank you again for your thoughtful engagement and constructive suggestions. We hope our responses and revisions have addressed your questions and strengthened the paper. We appreciated the opportunity to engage fruitfully with the reviewer. Given your positive assessment, we would be especially grateful if you could champion our paper during the reviewer discussion.
Thank you for your exceptionally thoughtful review and positive assessment of our work! We are delighted that you found our research question compelling, the presentation clear, and the cross-modal transfer results significant. We especially appreciate the time you took to provide detailed feedback beyond the requirements of the review form, and welcome the opportunity to engage with you on these broader scientific questions. We respond to your comments below.
Q1: Connection to theoretical ICL
We agree with this interpretation! Indeed, our approach is motivated by approximating Solomonoff induction, which fits naturally into the framework you describe. Under a prior over programs, the Bayes-optimal predictor infers a posterior over programs consistent with the observed context and uses this posterior to predict the next token. In principle, this encompasses inference over arbitrary computable generative processes, including HMMs, Gaussian processes, and more. Our KL regularization toward the uniform program prior is motivated precisely by this connection.
We also agree that learning a simple Markov approximation could already yield nontrivial performance on natural text. However, our ICL results in Section 3.2 provide evidence that the learner acquires richer predictive strategies. Tasks such as reverse string and stack require behavior beyond simple fixed-order Markov prediction, and models trained on the fixed universal prior or PCFGs largely fail on these tasks. Moreover, we have now evaluated a matched-size byte-level model pretrained on natural text: our self-play model matches its performance on associative recall and exceeds it on the other ICL evaluations.
We view these results as encouraging evidence that self-play can induce general-purpose inference strategies. Understanding precisely which inference algorithms emerge, and how they evolve with scale, is an exciting direction for future work.
Q2: Self-play with PCFG
We agree that incorporating domain-specific inductive biases could substantially improve practical performance. Indeed, the strong performance of PCFG pretraining on text and code suggests that hierarchical and compositional structure is particularly valuable for these domains.
While all Turing-complete programming languages can express the same class of computable data-generating processes, their inductive biases can matter substantially at finite compute. For example, probabilistic programming languages such as Church (Goodman et al., 2008) make stochastic and hierarchical structures easier to express, while learned libraries of reusable abstractions, as in DreamCoder (Ellis et al., 2021), can make complex compositional programs easier to discover.
In the present work, we deliberately use a minimal universal language to isolate the capabilities of a general-purpose learning procedure. We are nevertheless excited about richer program representations and improved search procedures, and view this as an important direction toward improving self-play pretraining.
Q3.2: Intuition for policy-gradient choices
Thank you for this suggestion. We will expand the explanation of the generator's training objective, particularly the motivation for the following components:
Reward normalization. The overall magnitude of the learning-progress reward changes substantially over training. We normalize rewards within each round because the relative differences between programs are more informative for updating the generator than changes in the overall reward scale. Relative rewards also preserve the correctness of the PG update due to the score function identity but can reduce variance of the gradient estimates; this is exactly the idea of control variates in statistics.
Importance sampling. Replayed programs were sampled from earlier generator policies. The importance ratio corrects for this distributional mismatch when estimating the policy gradient under the current generator.
We have incorporated these intuitions into the main text (see revised Methods section).