- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a transformer-based language model pretrained exclusively on synthetic sequences. Each sequence is generated by a newly sampled recurrent structural causal model (SCM). The resulting model predicts natural-language text and different families of deterministic sequences increasingly well as context grows, without any update of the weights.
Critical strengths
None.
Critical weaknesses
The paper does not sufficiently establish its contribution beyond existing theoretical understanding and empirical demonstrations of in-context learning. The theoretical ICL literature (e.g. this) already describes how next-token prediction over a distribution of data-generating processes can lead to a general inference strategy, capable of identifying the process underlying a new sequence. Moreover, Bloem has already demonstrated that transformers pretrained on synthetic sequences from random recurrent generators display nontrivial next-token prediction performance. The observation that synthetic pretraining enables nontrivial prediction of real text is therefore insufficient, by itself, to establish the contribution claimed here.
In this setting, an important reference predictor is the Bayesian posterior predictive induced by the proposed SCM prior, or a suitable approximation. Such a predictor could already achieve nontrivial performance on real text by exploiting statistical structure captured by this family of processes. The paper does not quantify this reference performance, making it difficult to assess how much of the reported behaviour follows from the chosen prior and how faithfully the transformer approximates the corresponding inference procedure.
Questions for authors
Several aspects of the construction and validation of the prior also require clarification.
-
At the end of page 5, the authors state that the statistical analysis uses the configuration that best matches the natural-language statistics, whereas the training configuration is selected for transfer to natural language. How were these configurations selected, and how do they differ?
-
Figure 3’s caption identifies the third panel as a bigram Zipf exponent, while the discussion describes the i.i.d. Zipf baseline as correcting the marginal distribution. How were the marginal exponent and other parameters of this baseline chosen, and how was the bigram exponent estimated?
-
How were the distributions over bigram and HMM parameters chosen? Their nearly uniform marginals appear to reflect these choices, rather than an inherent limitation of the model families. Since the SCM itself defines a hidden-state stochastic process, what features distinguish it from HMMs? Why were the baseline ensembles not chosen to match natural-language statistics, like the SCM?
-
How were the conditional entropies and mutual information in Figure 4 estimated? At higher n-gram orders, sparse counts can produce artificially low empirical conditional entropy, while finite-sample bias can produce a positive MI tail even for independent data. Please specify the estimators and any corrections, and establish whether the reported patterns persist with larger sample sizes.
4 comments
Selection of the two configurations
How were these configurations selected, and how do they differ?
Table 3 (Appendix A) now lists both configurations.
| Range | Statistics (Section 3) | Training |
|---|---|---|
| Nodes | 16 to 48 | 8 to 32 |
| Noise scale | 1/20 to 1/5 | 0 to 1/80 |
| Update layers | 1 to 2 | 1 to 6 |
| Parent-set size | 2 to 8 | 2 to 16 |
All other components are shared. The statistics configuration is the best of eight candidates under a score over ten statistics and the six languages. The training configuration was selected by transfer, using 54M-parameter models and a development set that is disjoint from the evaluation windows. A model trained on the statistics configuration reaches 3.13 bits per byte, compared with 2.70 for the tuned prior.
Zipf baseline and bigram exponent
... how was the bigram exponent estimated?
The exponent is fitted per sequence of 4,096 bytes by least squares on log frequency against log rank, over the upper half of the ranked byte bigrams (Appendix E). The i.i.d. Zipf baseline has a Zipfian distribution over single bytes. Its bigram frequencies are products of marginal probabilities, and their rank-frequency curve has a different slope from that of text. The parameters of all four baselines are sampled and not fitted to text.
Bigram and HMM baselines
How were the distributions over bigram and HMM parameters chosen? Their nearly uniform marginals appear to reflect these choices ...
In Section 3 the bigram and HMM baselines draw their parameters at random, without fitting to text, and their near-uniform marginals follow from that choice. They are not a limit of the two families.
The new Section 4.3 therefore compares the families in a different way. There the Markov and HMM priors draw each row from a Dirichlet distribution centred on a Zipf law (Appendix C), so their marginals are not uniform, and each family is judged by training a model on it. These models reach 3.60 and 4.14 bits per byte on Wikipedia, compared with 2.70 for the SCM prior.
SCM and HMM
Since the SCM itself defines a hidden-state stochastic process, what features distinguish it from HMMs?
You are right that both are hidden-state processes. They differ in the state and in how it evolves. An HMM has a finite set of discrete states with stochastic transitions between them. The SCM has a continuous state vector, in which each coordinate is updated by a structural equation of its parents in a sparse random graph, composed over several layers. The noise is small, so the dynamics are close to deterministic, and the emitted byte is a quantized reading of a few coordinates.
Estimators in Figure 4
How were the conditional entropies and mutual information in Figure 4 estimated? ... finite-sample bias can produce a positive MI tail even for independent data.
Both quantities are plug-in estimates on single sequences of 4,096 bytes, summarized over 500 sequences, and the mutual information uses the Miller-Madow correction.
Your concern about bias is justified. For English text with its bytes shuffled, the mutual information estimate is about 0.2 bits at every distance, so the values in Figure 4 at high orders and large distances are not estimates of the true quantities. The comparison remains like for like, because the prior and the six languages are measured with the same estimators at the same sequence length. Appendix E now specifies the estimators and states this limitation. We have not re-estimated on longer sequences in this revision.
This comment does not address my concern, in particular:
Existing theory explains why pretraining on a distribution of processes can produce in-context inference. It does not determine which priors transfer to natural text, or how far. The revised paper measures both.
All priors transfer to natural next. Figure 6 ranks some of those tried by the authors, but:
- The fact that Markov chains transfer more than HMMs makes me think that maybe the chosen HMM family was too simple, or perhaps the model was undertrained. If the model had learned to infer the best finite-dimensional HMM approximation of the dataset, it should perform better than one applying the best Markov-Chain approximation.
- I don't know what to make of the fact that the specific SCM prior considered here transfers better than others. The observation per se was already made, and, given my current understanding, I don't believe that this study adds much insight.
Thank you for the reply.
If the model had learned to infer the best finite-dimensional HMM approximation of the dataset, it should perform better than one applying the best Markov-Chain approximation.
That a family can express more does not make it a better prior for training a model to learn in context. What matters more is the learning signal in its samples.
I don't know what to make of the fact that the specific SCM prior considered here transfers better than others.
Unlike random recurrent networks, the SCM prior has an explicit structure that can be controlled: a causal graph, its fan-in and depth, the equation families, and the noise scale. In the tabular domain, this explicit structure has proven useful well beyond prediction: intervening on the graph gives training data for causal effect estimation, removing the effect of a protected attribute gives counterfactual fairness, and letting the graph change over time gives robustness to distribution shift. For sequences it has hardly been studied, and the paper provides a foundation for it.
That such transfer is possible in principle follows from theory. Whether a concrete prior achieves it on natural text, and how far, has to be shown, and this is what the paper does.
Thank you for the detailed questions. The revision cites and discusses Bloem (2025), specifies both prior configurations, and describes the estimators of Section 3. We respond to each point below.
Contribution
We have added Bloem (2025), which we had missed. The introduction now credits it, and Section 5 discusses it as the closest result to ours.
Our contribution is the prior. We take the structural causal models behind tabular prior-fitted networks (Hollmann et al., 2023) and make them recurrent, so that one sampled model generates a sequence, and we show that this yields a strong prior for learning the structure of sequences in context. The revised introduction states the contribution in these terms.
Existing theory explains why pretraining on a distribution of processes can produce in-context inference. It does not determine which priors transfer to natural text, or how far. The revised paper measures both. On Wikipedia in six languages, the cost falls to between 0.9 and 2.4 bits per byte at one million bytes of context, which is below an online bigram, gzip, and PPMd on average (new first row of Table 2). With the architecture and the training budget fixed, changing only the prior moves the cost at 4,096 bytes of context from 5.13 bits per byte for i.i.d. Zipf to 2.70 for the tuned SCM prior (new Section 4.3). More equation families, larger parent sets, and deeper updates each help, consistent with findings on priors in the tabular domain (Qu et al., 2026).
Bayesian posterior predictive
The posterior predictive is the right reference in principle. For this prior it is intractable: it ranges over a random graph, an equation family and coefficients for each node, firing rates, and a state trajectory that is observed only through quantization, and we know of no approximation whose error could be bounded. In prior-fitted training the network itself is the approximation.
Two parts of the question can be measured. How much of the behaviour follows from the prior is the subject of the new Section 4.3. How closely a trained network approaches the exact predictor can be measured on a prior that is simple enough to allow it, which we leave to future work.