acceptodds
Under review as a conference paper at ICLR 2027

Learning to Learn a Language

Abstract

We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior. Given a prefix of real text, it learns to predict the language in context with frozen weights, having never seen a word of any real language. Every training sequence is generated by a recurrent structural causal model drawn fresh from a distribution over such models. The model never sees the same language twice during training, so the only way to predict the continuation is to infer the language from the prefix. Samples from this prior share the statistical signatures of natural text: Zipfian frequencies, slow entropy-rate convergence, and long-range dependence. On Wikipedia in six languages, bits per byte fall from the uniform eight to between and at one million bytes of context. Given numerals instead of text, PFLM learns to count, to compare magnitudes, and to add approximately. It predicts deterministic sequences like Rudin–Shapiro or the prime indicator, and it compresses six non-text domains, from source code to speech, below gzip and PPMd. The model has not learned a language. It has learned to learn one.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
in-context learning, prior-fitted networks, synthetic pretraining, language modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vbqCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a transformer-based language model pretrained exclusively on synthetic sequences. Each sequence is generated by a newly sampled recurrent structural causal model (SCM). The resulting model predicts natural-language text and different families of deterministic sequences increasingly well as context grows, without any update of the weights.

Critical strengths

None.

Critical weaknesses

The paper does not sufficiently establish its contribution beyond existing theoretical understanding and empirical demonstrations of in-context learning. The theoretical ICL literature (e.g. this) already describes how next-token prediction over a distribution of data-generating processes can lead to a general inference strategy, capable of identifying the process underlying a new sequence. Moreover, Bloem has already demonstrated that transformers pretrained on synthetic sequences from random recurrent generators display nontrivial next-token prediction performance. The observation that synthetic pretraining enables nontrivial prediction of real text is therefore insufficient, by itself, to establish the contribution claimed here.

In this setting, an important reference predictor is the Bayesian posterior predictive induced by the proposed SCM prior, or a suitable approximation. Such a predictor could already achieve nontrivial performance on real text by exploiting statistical structure captured by this family of processes. The paper does not quantify this reference performance, making it difficult to assess how much of the reported behaviour follows from the chosen prior and how faithfully the transformer approximates the corresponding inference procedure.

Questions for authors

Several aspects of the construction and validation of the prior also require clarification.

  1. At the end of page 5, the authors state that the statistical analysis uses the configuration that best matches the natural-language statistics, whereas the training configuration is selected for transfer to natural language. How were these configurations selected, and how do they differ?

  2. Figure 3’s caption identifies the third panel as a bigram Zipf exponent, while the discussion describes the i.i.d. Zipf baseline as correcting the marginal distribution. How were the marginal exponent and other parameters of this baseline chosen, and how was the bigram exponent estimated?

  3. How were the distributions over bigram and HMM parameters chosen? Their nearly uniform marginals appear to reflect these choices, rather than an inherent limitation of the model families. Since the SCM itself defines a hidden-state stochastic process, what features distinguish it from HMMs? Why were the baseline ensembles not chosen to match natural-language statistics, like the SCM?

  4. How were the conditional entropies and mutual information in Figure 4 estimated? At higher n-gram orders, sparse counts can produce artificially low empirical conditional entropy, while finite-sample bias can produce a positive MI tail even for independent data. Please specify the estimators and any corrections, and establish whether the reported patterns persist with larger sample sizes.

4 comments

Thank you for the detailed questions. The revision cites and discusses Bloem (2025), specifies both prior configurations, and describes the estimators of Section 3. We respond to each point below.

Contribution

Bloem has already demonstrated that transformers pretrained on synthetic sequences from random recurrent generators display nontrivial next-token prediction performance.

We have added Bloem (2025), which we had missed. The introduction now credits it, and Section 5 discusses it as the closest result to ours.

Our contribution is the prior. We take the structural causal models behind tabular prior-fitted networks (Hollmann et al., 2023) and make them recurrent, so that one sampled model generates a sequence, and we show that this yields a strong prior for learning the structure of sequences in context. The revised introduction states the contribution in these terms.

Existing theory explains why pretraining on a distribution of processes can produce in-context inference. It does not determine which priors transfer to natural text, or how far. The revised paper measures both. On Wikipedia in six languages, the cost falls to between 0.9 and 2.4 bits per byte at one million bytes of context, which is below an online bigram, gzip, and PPMd on average (new first row of Table 2). With the architecture and the training budget fixed, changing only the prior moves the cost at 4,096 bytes of context from 5.13 bits per byte for i.i.d. Zipf to 2.70 for the tuned SCM prior (new Section 4.3). More equation families, larger parent sets, and deeper updates each help, consistent with findings on priors in the tabular domain (Qu et al., 2026).

Bayesian posterior predictive

... an important reference predictor is the Bayesian posterior predictive induced by the proposed SCM prior, or a suitable approximation.

The posterior predictive is the right reference in principle. For this prior it is intractable: it ranges over a random graph, an equation family and coefficients for each node, firing rates, and a state trajectory that is observed only through quantization, and we know of no approximation whose error could be bounded. In prior-fitted training the network itself is the approximation.

Two parts of the question can be measured. How much of the behaviour follows from the prior is the subject of the new Section 4.3. How closely a trained network approaches the exact predictor can be measured on a prior that is simple enough to allow it, which we leave to future work.

Selection of the two configurations

How were these configurations selected, and how do they differ?

Table 3 (Appendix A) now lists both configurations.

RangeStatistics (Section 3)Training
Nodes16 to 488 to 32
Noise scale1/20 to 1/50 to 1/80
Update layers1 to 21 to 6
Parent-set size2 to 82 to 16

All other components are shared. The statistics configuration is the best of eight candidates under a score over ten statistics and the six languages. The training configuration was selected by transfer, using 54M-parameter models and a development set that is disjoint from the evaluation windows. A model trained on the statistics configuration reaches 3.13 bits per byte, compared with 2.70 for the tuned prior.

Zipf baseline and bigram exponent

... how was the bigram exponent estimated?

The exponent is fitted per sequence of 4,096 bytes by least squares on log frequency against log rank, over the upper half of the ranked byte bigrams (Appendix E). The i.i.d. Zipf baseline has a Zipfian distribution over single bytes. Its bigram frequencies are products of marginal probabilities, and their rank-frequency curve has a different slope from that of text. The parameters of all four baselines are sampled and not fitted to text.

Bigram and HMM baselines

How were the distributions over bigram and HMM parameters chosen? Their nearly uniform marginals appear to reflect these choices ...

In Section 3 the bigram and HMM baselines draw their parameters at random, without fitting to text, and their near-uniform marginals follow from that choice. They are not a limit of the two families.

The new Section 4.3 therefore compares the families in a different way. There the Markov and HMM priors draw each row from a Dirichlet distribution centred on a Zipf law (Appendix C), so their marginals are not uniform, and each family is judged by training a model on it. These models reach 3.60 and 4.14 bits per byte on Wikipedia, compared with 2.70 for the SCM prior.

SCM and HMM

Since the SCM itself defines a hidden-state stochastic process, what features distinguish it from HMMs?

You are right that both are hidden-state processes. They differ in the state and in how it evolves. An HMM has a finite set of discrete states with stochastic transitions between them. The SCM has a continuous state vector, in which each coordinate is updated by a structural equation of its parents in a sparse random graph, composed over several layers. The noise is small, so the dynamics are close to deterministic, and the emitted byte is a quantized reading of a few coordinates.

Estimators in Figure 4

How were the conditional entropies and mutual information in Figure 4 estimated? ... finite-sample bias can produce a positive MI tail even for independent data.

Both quantities are plug-in estimates on single sequences of 4,096 bytes, summarized over 500 sequences, and the mutual information uses the Miller-Madow correction.

Your concern about bias is justified. For English text with its bytes shuffled, the mutual information estimate is about 0.2 bits at every distance, so the values in Figure 4 at high orders and large distances are not estimates of the true quantities. The comparison remains like for like, because the prior and the six languages are measured with the same estimators at the same sequence length. Appendix E now specifies the estimators and states this limitation. We have not re-estimated on longer sequences in this revision.

This comment does not address my concern, in particular:

Existing theory explains why pretraining on a distribution of processes can produce in-context inference. It does not determine which priors transfer to natural text, or how far. The revised paper measures both.

All priors transfer to natural next. Figure 6 ranks some of those tried by the authors, but:

  1. The fact that Markov chains transfer more than HMMs makes me think that maybe the chosen HMM family was too simple, or perhaps the model was undertrained. If the model had learned to infer the best finite-dimensional HMM approximation of the dataset, it should perform better than one applying the best Markov-Chain approximation.
  2. I don't know what to make of the fact that the specific SCM prior considered here transfers better than others. The observation per se was already made, and, given my current understanding, I don't believe that this study adds much insight.

Thank you for the reply.

If the model had learned to infer the best finite-dimensional HMM approximation of the dataset, it should perform better than one applying the best Markov-Chain approximation.

That a family can express more does not make it a better prior for training a model to learn in context. What matters more is the learning signal in its samples.

I don't know what to make of the fact that the specific SCM prior considered here transfers better than others.

Unlike random recurrent networks, the SCM prior has an explicit structure that can be controlled: a causal graph, its fan-in and depth, the equation families, and the noise scale. In the tabular domain, this explicit structure has proven useful well beyond prediction: intervening on the graph gives training data for causal effect estimation, removing the effect of a protected attribute gives counterfactual fairness, and letting the graph change over time gives robustness to distribution shift. For sequences it has hardly been studied, and the paper provides a foundation for it.

That such transfer is possible in principle follows from theory. Whether a concrete prior achieves it on natural text, and how far, has to be shown, and this is what the paper does.

Reviewer_3puaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper seeks to train a byte-level language model that can learn real languages in context despite being pretrained entirely on synthetic, non-linguistic data. The authors construct a distribution over recurrent structural causal models, each of which defines a synthetic language. Importantly, the authors design this distribution to reproduce three characteristics of natural text: Zipfian frequencies, slow entropy-rate convergence, and long-range dependencies. At training time, the authors instantiate a 300M-parameter transformer, sampling a new language and corresponding sequence for each training example. The resulting model exhibits a range of capabilities, including basic numerical reasoning, prediction of deterministic sequences, and compression improvements over several non-text domains. As the title suggests, the authors also find that the resulting model can predict natural languages in context across six languages, significantly outperforming a uniform baseline after one million bytes of context.

Critical strengths

Put simply, I find this paper's central idea extremely cool. Whereas many papers focus on incremental improvements to existing training recipes, it is refreshing to see researchers investigate a more fundamental question about how general learning capabilities emerge. Concretely, the breadth of transfer to natural language, despite the complete absence of natural-language text during pretraining, is genuinely surprising and potentially significant.

More broadly, I think this work gets at an important question: how much structure does pretraining data actually need to induce general learning capabilities? I would encourage the authors to emphasize this question more, as I think it has substantial implications beyond the particular construction studied here.

Critical weaknesses

I really want to love this paper, but the authors forgo a fair amount of experimental rigor, making it difficult to assess the strength and implications of their findings.

The paper provides little evidence that the model has actually learned to learn a language

  • The primary natural language results demonstrate a reduction in cross-entropy under teacher-forced, (and not closed loop) evaluation. There are no text samples or generations from the model itself. Can it produce coherent sentences or paragraphs after sufficient context? Does it learn syntactic or semantic structure? If the answer is no, I don't think this necessarily undermines the value of the work, but the authors should be forthcoming about these limitations.
  • There are no meaningful baselines on the natural-language benchmarks beyond uniform prediction. English reaches approximately 2 bits per byte (I believe), which is substantially worse than results reported by conventional language models trained on text. Again, this is understandable given the training setup, but comparisons against conventional language models and adaptive compression methods would help contextualize the result.
  • The numerical tasks are fairly limited; this doesn’t bother me. I am particularly curious whether the model acquires any compositional or systematic reasoning capabilities, rather than simply exploiting statistical regularities in the demonstrated examples. If the main claim is that the model has learned to learn a language, I cannot recommend acceptance without substantially more evidence.

The authors present a sophisticated synthetic data distribution, but make little attempt to isolate which properties are responsible for downstream transfer.

  • I would appreciate a more thorough discussion of how the authors arrived at this construction, including what worked and what didn't. This would be considerably more informative to the community than simply presenting a working construction.
  • The paper should include ablations of the pretraining distribution itself. For example, what happens if a model is pretrained on data with Zipfian frequencies and slow entropy-rate convergence, but without long-range dependencies? What about simpler generators, such as Markov chains or HMMs? The current experiments compare the statistical properties of these distributions, but not the capabilities of models pretrained on them.
  • More fundamentally, the paper does not establish why the three cited statistical characteristics capture sufficient structure for transfer to natural language.

In my opinion, the most interesting question raised by this paper is what minimal structure a pretraining distribution needs to induce different learning capabilities. I would encourage the authors to consider framing the work around this question, rather than the stronger claim of learning natural languages in context.

Questions for authors

Can the authors address the points in my weaknesses section? Again, this is genuinely one of the more interesting papers I’ve encountered, but it needs substantially more rigor before I can recommend acceptance. I’m very open to changing my score if the authors can adequately address my concerns.

2 comments

Thank you for the detailed and encouraging review. The revision adds the ablations, text samples, and baselines you asked for, and relates what we find about the prior to what is known from the tabular domain.

Text samples

There are no text samples or generations from the model itself. Can it produce coherent sentences or paragraphs after sufficient context?

We have added samples in Appendix F (Figure 9). For each of the six languages, the model reads 256,000 bytes of the first evaluation window and continues it by greedy decoding.

The continuations remain in the language and script of the context and consist of real words in a locally plausible order. They follow the format of the text, for example a dated list of people in the Korean window, but not its content, and greedy decoding leads to repetition. The model does not produce coherent paragraphs, and we have no evidence that it learns semantic structure. We state this in the conclusion and in Appendix F.

Baselines on natural language

There are no meaningful baselines on the natural-language benchmarks beyond uniform prediction.

Table 2 now includes natural language. Averaged over the six languages on windows of one million bytes, the online bigram reaches 3.34 bits per byte, gzip 2.74, PPMd 1.88, and PFLM 1.70.

Your estimate for English, about 2 bits per byte at one million bytes, is correct. Language models trained on text perform far better, and the conclusion states this.

Compositional reasoning

I am particularly curious whether the model acquires any compositional or systematic reasoning capabilities ...

This is indeed an interesting question, and our evidence on it is so far limited. In the number tasks, no query numeral appears among the in-context examples, so the model has to apply what it inferred to new numerals. We have not yet run an evaluation designed for compositional generalization.

Ablations of the pretraining distribution

The paper should include ablations of the pretraining distribution itself. ... What about simpler generators, such as Markov chains or HMMs?

The new Section 4.3 reports these experiments. We train a 54M-parameter model on each prior, with 4,096 bytes of context and 5 billion tokens, and evaluate it on Wikipedia in the six languages.

Pretraining priorBits per byte
i.i.d. Zipf5.13
Markov chain3.60
Hidden Markov model4.14
SCM, linear equations3.01
SCM, nonlinear equations2.83
SCM, tuned2.70

The model trained on i.i.d. Zipf stops improving after a few hundred bytes of context, whereas the models trained on the SCM prior are still improving at the end of the window (Figure 6).

What worked and what did not

... including what worked and what didn't.

Three changes improved transfer: nonlinear structural equations, larger parent sets, and more than one update layer per step. Limiting each node to two parents increases the cost by 0.27 bits per byte, and a single update layer increases it by 0.15 (Figure 7).

Selecting the prior by its statistical match to text did not help. That configuration reaches 3.13 bits per byte, which is worse than the SCM with only linear equations.

The number of nodes and the noise scale change the result by at most 0.10. For multirate firing, models trained with and without it differ by at most 0.03. A context of 4,096 bytes cannot show a long-range benefit, so we consider this comparison inconclusive and do not report it in the paper.

Sufficiency of the three statistics

... the paper does not establish why the three cited statistical characteristics capture sufficient structure for transfer.

We agree with you that they are not sufficient. The ablation shows that a closer statistical match does not improve transfer. In the revision, Section 3 describes the samples of the prior, and the prior itself is selected by transfer (Appendix A).

Framing

I would encourage the authors to consider framing the work around this question ...

We have moved the paper in this direction. The introduction now states the contribution in terms of the prior: the structural causal models behind tabular prior-fitted networks (Hollmann et al., 2023), made recurrent, yield a strong prior for learning the structure of sequences in context. The new Section 4.3 examines which parts of the prior the transfer depends on.

This places our ablations next to what the tabular domain has established, that priors with richer and more diverse structure transfer better to real data. TabICLv2, for example, reports that its new prior, which has more types of random functions, accounts for its largest gain (Qu et al., 2026). We find the same for sequences: more equation families, larger parent sets, and deeper updates each improve transfer to text (Section 4.3). What the minimal structure is remains open. Our ablations, with one training run per prior at 54M parameters, are a first step toward that question.

Thank you for your response. I will move my score to 3.

Reviewer_zt2AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a Prior-Fitted Language Model, a ~300M-parameter byte-level model trained on sequences generated by a synthetic prior over recurrent structural causal models. Each training sequence uses a newly sampled generator, encouraging the network to adapt its predictions to the observed prefix. The model is trained on 150B synthetic tokens with a context-length curriculum reaching approximately one million bytes.

With frozen weights, PFLM improves next-byte prediction on Wikipedia in six languages as context increases. Additional experiments evaluate numerical tasks, deterministic sequences, and compression across six non-text domains. The paper also compares statistical properties of synthetic sequences with natural text. Its central contribution is demonstrating transfer from synthetic process prediction to in-context prediction of real data without natural-language gradient training.

Critical strengths

Training exclusively on randomly generated processes and obtaining useful prediction on natural text with frozen weights is a meaningful extension of prior-fitted sequence modeling. The setup separates exposure to natural language during gradient training from adaptation through context.

The numerical tasks, deterministic sequences, and compression results across code, molecules, biological sequences, music, and audio suggest that the learned predictor transfers across multiple kinds of structure. The reported improvements over PPMd in the non-text domains are particularly relevant evidence.

Critical weaknesses

Figure 1 compares natural-language prediction with the uniform eight-bit rate, but adaptive statistical predictors also improve substantially as they observe a stream. The online bigram, PPMd, and other compression baselines should be evaluated on the same Wikipedia windows and context budgets. Without these comparisons, the results do not establish how much the learned prior contributes beyond conventional adaptation to byte frequencies, local dependencies, and repetition.

Section 3 states that the statistical-fingerprint configuration differs from the training configuration, which was selected using transfer to natural language. This means the fingerprint results do not directly characterize the distribution used to train the evaluated model. There are also no model-level transfer comparisons against simpler synthetic priors or ablations of the multi-rate dynamics. The paper should explain the real-text development data and selection procedure, establish separation from final evaluation, and test which prior components actually improve transfer.

Questions for authors

How does PFLM compare with PPMd and adaptive n-gram or context-mixing predictors on the same Wikipedia streams? Please distinguish local prediction cost near a given context length from cumulative compression cost over the entire stream.

1 comment

Thank you for the constructive comments. The revised paper contains the baselines and the ablations you asked for. We respond to each point below.

Compression baselines on Wikipedia

The online bigram, PPMd, and other compression baselines should be evaluated on the same Wikipedia windows and context budgets.

We have added these baselines to Table 2. Averaged over the six languages on windows of one million bytes, the online bigram reaches 3.34 bits per byte, gzip 2.74, PPMd 1.88, and PFLM 1.70.

Local and cumulative cost

Please distinguish local prediction cost near a given context length from cumulative compression cost over the entire stream.

The values above are cumulative costs over the whole window. In the local cost, the bits per byte near a given context length, PFLM is below PPMd at every length we measured. The margin decreases with context, from 0.67 bits at 256 bytes to 0.26 at 36 KB and 0.09 at 800 KB, because PPMd continues to adapt to the document. We have not evaluated a context-mixing compressor, which we expect to be stronger than PPMd.

Statistics and training configurations

... the fingerprint results do not directly characterize the distribution used to train the evaluated model.

You are right. The revised paper lists both configurations in Table 3 (Appendix A) and describes how each was selected. They differ in four ranges: the number of nodes, the noise scale, the number of update layers, and the parent-set size.

The training configuration was selected with 54M-parameter models on a development set of 64 windows of 4,096 bytes per language, which is disjoint from the evaluation windows. The revised paper now describes this procedure in Appendix A. We also trained a model on the statistics configuration and report the result in the revised paper. It reaches 3.13 bits per byte, compared with 2.70 for the tuned prior, so the closest statistical match is not the best prior for transfer (Section 4.3).

Simpler priors and components of the prior

There are also no model-level transfer comparisons against simpler synthetic priors or ablations of the multi-rate dynamics.

The new Section 4.3 reports these experiments. We train a 54M-parameter model on each prior, with 4,096 bytes of context and 5 billion tokens, and evaluate it on Wikipedia in the six languages.

Pretraining priorBits per byte
i.i.d. Zipf5.13
Markov chain3.60
Hidden Markov model4.14
SCM, linear equations3.01
SCM, nonlinear equations2.83
SCM, tuned2.70

Among the hyperparameter ranges of the prior (Figure 7), limiting each node to two parents increases the cost by 0.27 bits per byte and a single update layer increases it by 0.15. The number of nodes and the noise scale change it by at most 0.10.

Regarding the multirate dynamics, models trained with and without them differ by at most 0.03 bits per byte. A context of 4,096 bytes cannot show a long-range benefit, so we consider this comparison inconclusive and do not report it in the paper.

Each prior was trained once, and we therefore interpret only the large differences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.