acceptodds
Under review as a conference paper at ICLR 2027

Improving Bayesian In-Context Learning in Markov Transformers

Abstract

A transformer can enter a generalizing in-context learning regime while remaining far from the corresponding optimal predictor. What prevents it from closing this residual gap? We study this question in Markov-chain prediction, where the Bayes posterior is available exactly and the learned attention circuit can be inspected. We introduce Symmetry-Aware In-context Pre-training (SAIP) for two-block transformers. SAIP guides the formation of previous-token attention early in training and lightly regularizes predictions toward the closed-form posterior throughout training. Inference uses the two-block model without evaluating auxiliary losses. Across 18 task-pool/seed settings, SAIP reduces mean KL divergence to the bigram posterior by 89.3%, with at least a 69.8% reduction in every setting. It also strengthens previous-token attention and moves the empirical generalization boundary to smaller task pools. Ablations attribute most of the accuracy gain to posterior regularization, while attention measurements reveal a sharper previous-token circuit. These results show how training can improve the accuracy of Bayesian in-context prediction while changing the attention circuit that implements it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Bayesian In-Context Learning, Induction Heads, Markov Chains

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dSkLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper aims to show, through a controlled experiment, that a transformer can adopt the right inference strategy for a given task through ICL without necessarily making accurate predictions. Moreover, it suggests that training can improve accuracy when guided by an appropriate auxiliary task loss. Following the Markov-ICL benchmark and two-block transformer of Gibson et al. (2026), the paper uses simple Markov sequences, four Bayesian predictors covering memorization and generalization, and a simplified two-layer transformer. What is added to Gibson et al. is a training recipe called SAIP; small attention modification, an attention auxiliary loss, and a posterior-matching loss that directly supervises the generalizing Bayesian predictor. The experiments show that SAIP drives generalization at smaller task-pool sizes and improves accuracy, with most of the accuracy gain coming from the posterior-matching loss.

Critical strengths
  • The paper contains ample simulation and ablation results showing SAIP’s ability to improve accuracy on the Gibson Markov-ICL benchmark.
  • The improvement from the posterior loss after the phase transition is illustrative. It may help us think about how auxiliary losses, commonly used in deep learning methods such as REPA, improve performance.
Critical weaknesses
  • The biggest weakness is its scope. The idea that transformer ICL may adopt the right inference strategy without implementing it accurately seems plausibly relevant to ICL in pretrained LLMs. However, the current experiments are so limited that the conclusion is largely confined to improving accuracy on the Gibson Markov-ICL benchmark. What can be established so far is that, at least for this specific benchmark task, auxiliary posterior supervision helps. Extending this conclusion would require additional experiments in more general settings, including open-source LLMs. Prior-fitted network (PFN) pretrained with synthetic task could also be informative.

  • Second, the proposed SBI attention modification has an intuitive motivation, but the ablations show that most of the accuracy improvement comes from the posterior loss. This leaves limited evidence that SBI itself provides a meaningful additional benefit to prediction accuracy.

  • Finally, as the authors acknowledge, the posterior loss requires a known, tractable posterior. This is available in the synthetic benchmark, but its applicability to more realistic tasks remains unclear.

Questions for authors

Q1. I wonder whether “learning an inference strategy but implementing it inaccurately” is representative of real-world tasks or mainly arises in Gibson et al.’s setting. It is often argued that neural networks generalize by learning algorithms shared across tasks [1, 2]. Similarly, the literature on ICL [3, 4] studies how transformers can implement underlying inference algorithms, such as gradient descent and Bayesian prediction. How do the current findings relate to this broader view, and what evidence suggests that the observed gap extends beyond this benchmark?

[1] Xu, K., Li, J., Zhang, M., Du, S. S., Kawarabayashi, K.-i., and Jegelka, S. (2020). What Can Neural Networks Reason About? ICLR

[2] Veličković, P., and Blundell, C. (2021). Neural Algorithmic Reasoning. Patterns

[3] Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. (2023). Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning. NeurIPS

[4] Kang, G., Lee, C. J., and Cheng, X. (2026). Transformers Can Learn Posterior Predictive Distributions In-Context. ICML

Q2. Relatedly, can the authors explain how the current results extend to more general settings, such as next-token prediction in LLM pretraining or PFN for tabular prediction, and provide evidence of their relevance? If so, what implications do these results have for pretraining or posttraining, architectural modifications, or controlled interventions?

11 comments

Part 1/3 — New experiments: open-source LLMs and a PFN setting

We thank the reviewer for a constructive review and for the concrete suggestion to test open-source LLMs and PFNs; we ran both. Unless stated otherwise, numbers are means over seeds, evaluated on fresh tasks drawn from the prior rather than on the training pool, from a clean re-implementation (see Part 3).

(a) Frozen pretrained LLMs, purely in context. Qwen2.5-0.5B / 1.5B / 7B read Markov sequences as comma-separated digits (V=10, T=128, Dirichlet(1) rows, fresh matrices; 1024 sequences, 2 seeds). We use the next-symbol distribution over the ten digit tokens, which carry >99% of the probability mass.

modelKL(2-Gen‖LLM), t=128KL(1-Gen‖LLM), t=128KL(2-Gen‖LLM), t=32
0.5B0.1380.0400.085
1.5B0.1170.0510.074
7B0.1840.0660.083
  • For comparison, two-block transformers trained on this task reach 0.018–0.033 with CE and 0.004 with SAIP.
  • The LLMs do learn in context: KL to 2-Gen falls from ≈0.41–0.46 at t=2 to ≈0.08 at t≈32.
  • Beyond that point they move toward in-context unigram frequencies rather than the bigram posterior. The gap grows with context length and does not shrink with scale.

This is the "strategy learned but executed inaccurately" phenomenon, now in pretrained LLMs.

(b) Fine-tuning a pretrained LLM on a finite task pool. Full fine-tuning of Qwen2.5-0.5B (2000 steps, batch 32, lr 3e-5, 3 seeds):

KCECE + exact 2-Gen anchorCE + learned-teacher anchor
320.250 (2-Mem nearest)0.00230.0092
1280.098 (2-Mem nearest)0.00050.0065
20480.00110.00010.0056
∞0.00070.00010.0054
  • The teacher is a small CE-only transformer trained on unlimited synthetic tasks; it never sees the closed form.
  • At K=128, CE first approaches the generalising solution (0.023 at step 500), then drifts to the memorising one (0.098 at step 2000). This is the transient pattern seen in small models, and both anchors prevent it.
  • Qwen2.5-1.5B (K=128, 3 seeds) shows the same pattern: CE drifts to memorisation (0.102, 2-Mem nearest), while the exact anchor gives 0.0005 and the learned-teacher anchor 0.0066.
  • One honest difference from small models: with abundant tasks (K≥2048), the pretrained LLM is already accurate under CE. For pretrained models, the benefit of the auxiliary target therefore lies in generalising from limited task diversity, not in closing a gap that persists at large K.

(c) PFN-style in-context Bayesian linear regression, the setting of Raventós et al. (2023). Tasks are w∼N(0,I8)w\sim\mathcal N(0,I_8) drawn from a finite pool of K tasks, with 32 noisy (x, y) pairs per sequence. The model is a 4-layer transformer with a Gaussian output head trained by NLL, run with 5 seeds. The metric is KL(posterior predictive ‖ model).

KNLLNLL + exact PPD anchorNLL + learned-teacher anchor
323.48 (dMMSE nearest)0.270.23
1280.0740.0240.024
5120.0240.0170.017
20480.0200.0150.016
∞0.0190.0140.015
  • The NLL-trained PFN stays measurably away from the posterior predictive even with unlimited tasks (0.019).
  • A posterior-predictive auxiliary target reduces this gap at every K.
  • A learned teacher, i.e. an NLL-only PFN trained on the full prior, is as effective as the exact target.
  • At K=32, both anchors switch the solution from pool-memorising (dMMSE) to ridge-like, but the error on new tasks remains large.

Part 2/3 — Q1 (representativeness and the algorithmic view) and Q2 (implications)

Q1. Is "right strategy, inaccurate execution" specific to Gibson et al.'s setting? The new evidence suggests it is not:

  • the gap appears in frozen pretrained LLMs and is not removed by scale (Part 1a);
  • when an LLM is fine-tuned on a finite task pool, CE first reaches and then leaves the generalising solution (1b);
  • in regression, an NLL-trained PFN stays measurably away from the posterior predictive (1c).

Relation to the algorithmic view.

  • Existence vs. what training finds. Ahn et al. (2023) and Kang, Lee & Cheng (2026) show that transformers can implement, in context, preconditioned gradient descent or the posterior predictive of GP regression. These results characterise what a transformer implements at an optimum, a critical point or an explicit construction, typically with unlimited task samples. Our results concern whether standard training finds an accurate implementation when task diversity is finite. In our setting it does not, although an accurate solution exists within the same architecture (0.0042 vs 0.033 at K=128).
  • Algorithmic alignment. Xu et al. (2020) relate sample efficiency to how well an architecture's structure matches the algorithm. The two-block architecture matches the count-and-normalise algorithm well enough to represent it accurately, yet from a finite pool CE selects a less accurate solution. Alignment governs what is learnable; it does not by itself determine which of several fitting solutions training selects.
  • Supervising the computation. Neural algorithmic reasoning (Veličković & Blundell, 2021) often supervises intermediate algorithmic steps. Our anchor is an output-level analogue. Notably, directly supervising the intermediate circuit does not improve accuracy: hard-coded previous-token attention (φ=1) gives 0.032 vs CE 0.033 at K=128 and is unstable at K=2048. What helps is a target that generalises better than the student, whether exact or learned.

Q2. Implications, stated as testable claims.

  1. Pre-training on synthetic priors (PFNs). When a PFN is trained on a finite set of prior samples, or fine-tuned on a narrower family, an auxiliary distillation target from a generalist trained on the full prior mitigates drift toward the finite pool. This holds in our Markov, regression and LLM fine-tuning results, at the cost of one teacher forward pass per batch.
  2. Post-training / fine-tuning on narrow task families. The pattern of transient generalisation followed by memorisation appears when fine-tuning a pretrained LLM (1b). Monitoring, and anchoring to, the divergence from a generalising reference is a cheap control.
  3. Architectural or attention interventions. Imposing the "correct" attention pattern did not improve accuracy in any setting we tried, and the SBI bias contributes nothing measurable (see below). We therefore do not recommend architectural priors as the lever, and we will report this as a negative result.
  4. Controlled interventions. The anchor must stay active. Removing it at mid-training loses most of the gain (K=128: SAIP 0.0042 → 0.022; anchor alone 0.0081 → 0.028), while switching it on only at mid-training recovers it (0.0056). It acts as a continual regulariser, not as an early "circuit unlock".

Scope of these claims. We did not run LLM pre-training or tabular PFNs (e.g. TabPFN); Bayesian linear regression is the canonical PFN prior we could test within the rebuttal period. The claims above therefore concern distillation under limited task diversity, not pre-training at scale.

SBI. We agree that SBI gives no meaningful benefit. With SBI removed entirely (absent, not merely zero-initialised), results are 0.0056 / 0.0045 / 0.0042 at K=32 / 128 / 2048, versus SAIP 0.0055 / 0.0042 / 0.0040; the same holds at widths 128 and 256. We will remove SBI from the proposed recipe and present the method as CE + posterior anchor, plus a small PTA gain (0.0053 → 0.0042 at K=128).

A known, tractable posterior. Our main addition is the learned-teacher anchor: a CE-only model trained on unlimited synthetic tasks, with no closed form anywhere in the pipeline.

  • In the Markov setting it gives 0.0117 / 0.0100 / 0.0090 at K=32 / 128 / 2048, versus CE 0.253 / 0.033 / 0.018.
  • In PFN regression it matches the exact target.
  • The teacher must generalise better than the student but need not be exact. A mis-specified prior (Dirichlet α=0.3 or 3) is no better than CE at K=128 and worse at K=2048, and an EMA self-teacher collapses.

Part 3/3 — Corrections and revision plan

While re-implementing the pipeline for the rebuttal we fixed several issues. All numbers in this response use the corrected code:

  1. the evaluation's 2-Gen reference omitted the latest transition;
  2. the Mem references indexed the true task instead of using the posterior mixture;
  3. evaluation used only the training pool (we now also report fresh matrices);
  4. the anchor KL was summed over the 128 positions, so λ=0.05 is a per-token weight of 6.4, not a "light" regulariser;
  5. the ablations "without SBI" kept a trainable bias.

Size of the improvement. With 5 seeds and 9 pool sizes, SAIP reduces KL to the 2-Gen predictor by 92.7% on fresh matrices (equal-weight mean 85.1%; minimum 70.4%), consistent with the submitted 89.3%. Against the true transition matrix, which the anchor never sees, the excess risk over the Bayes predictor falls from 0.032 to 0.0041 nats at K=128. With stationary initialisation the count predictor is within KL 0.0002 of the exact Bayes predictive, and a uniform-initialisation control gives the same results.

Further robustness.

  • Model size (3 seeds): at K=128, CE / SAIP give 0.051 / 0.0039 at width 128, 0.093 / 0.0033 at width 256, and 0.087 / 0.0024 with 4 blocks.
  • Second-order chains (K=128): the full recipe with a trigram anchor gives 0.0034 (anchor alone 0.011) versus 0.050 for CE; a bigram anchor fails (0.116).
  • LLM input format: without comma separators the frozen LLMs show the same picture (KL(2-Gen‖LLM) 0.15–0.19 at t=128; digit-token mass 0.94–0.96).
  • Anchor weight (3 seeds, K=128): SAIP improves monotonically over λ ∈ [0.005, 0.2] and saturates beyond 0.05 (0.018 → 0.0042 → 0.0037).

Revision plan.

  • Reframe the paper around the transferable findings: the gap under finite task diversity, the dissociation of strategy, circuit and accuracy, and anchoring to a generalist teacher.
  • Add the LLM and PFN experiments as a new section.
  • Present the learned-teacher anchor as the practical, non-oracle version of the method.
  • Drop SBI from the recipe.
  • Report all corrections and the uniform-initialisation control.

We hope these experiments address the reviewer's questions on scope and relevance, and we would be glad to discuss further.

Reviewer_d2z2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on prior works (Park et al. (2025) and Gibson et al. (2026)), which investigate different in-context algorithmic strategies learned by transformers trained on Markov chains, depending on the number of distinct chains observed during training and the training duration. These works show that models either memorize the training chains or generalize to the in-context chain, favoring different Bayes-optimal predictors across training regimes. The current paper starts from the observation that even in the generalizing regime, where the transformer is closer to the generalizing solution, the attained error remains far from zero. The authors aim to reduce this error by introducing regularization terms into the training objective that encourage the expected induction-head circuit and closeness to the Bayes-optimal predictor.

Critical strengths

The problem setup is timely, given the recent interest in studying the training dynamics and performance of transformers under the next-token prediction objective on controlled tasks such as Markov chains. The paper also employs a variety of empirical analyses and training interventions to investigate the problem in more depth.

Critical weaknesses

I have two main concerns with the paper.

  1. Presentation and clarity: The paper is not written very carefully.

    • Some notation is non-standard or not clearly defined. To give a few examples, πθ\pi^\theta is never defined in the text, and K1∗K_1^* is not clearly defined beyond being described as the transition to generalization. However, in Figure 2(b), the line corresponding to K1∗K_1^* does not appear to indicate any visible transition in behavior. Similarly, the notation used for the memorizing predictors in lines 140–144 is difficult to understand without familiarity with the previous works.

    • Some methodological details also appear to be missing. Based on the setup in previous works, I assume that the test sequences are generated from transition matrices other than P1,…,PKP_1,\ldots,P_K used during training, but I do not believe this is explicitly stated anywhere. I also think that each baseline used in the paper should be briefly introduced to make the paper self-contained. For instance, the paper discusses Posterior Distillation and cites Hahn & Goyal (2023), but I could not identify the exact training procedure or objective for this method, nor could I find its connection to the cited work.

    • The paper presents many different analyses, which could be a strength. However, in its current form, the results are difficult to connect into a coherent flow of arguments. Moreover, several analyses are presented without sufficient methodological details or clear motivation and contextualization. For example, in the paragraph around line 358, I find it unclear why the authors study linear interpolation of the parameters, what is meant by "previous token identity," or how to interpret the statement "Stronger organization of the intermediate representation therefore accompanies improved final prediction."

    • The introduction and conclusion contain several broad statements, such as "Our controlled setting exposes a separation between strategy selection, estimator accuracy, and circuit structure." However, I find it difficult to see how such claims are grounded in the presented results.

  2. Methodology: My main concern is with the proposed methodology. The central questions in prior works studying the learning of Markov chains are to identify under what conditions a model trained with next-token prediction can recover a Bayes-optimal predictor and whether the learned circuits are interpretable. However, the improvements achieved by the proposed method rely on assuming access to the exact Bayes-optimal predictor or circuit that the model is supposed to learn.

    In particular, the method effectively provides the optimal target predictor through a regularization term in the training objective. If the learning algorithm is assumed to have access to this ground-truth predictor, one could simply use it directly for prediction. This leaves me confused about what question the proposed methodology is intended to answer. From this perspective, the fact that most of the improvement comes from the 2G regularization term also seems rather expected.

    Similarly, the two attention-related interventions, SBI and PTA, essentially enforce the same inductive bias. Again, if we assume that the desired attention pattern is already known to be optimal, why not simply impose this pattern directly?

Questions for authors

See above

6 comments

Part 1/3 — Methodology

We thank the reviewer for the detailed review. We first address the methodological concern, then the presentation. Unless stated otherwise, all numbers are KL(2-Gen ‖ model) at the final position on fresh transition matrices, averaged over 5 seeds, from a clean re-implementation (corrections listed in Part 3).

"If the learner has access to the ground-truth predictor, why not use it for prediction?" We agree that the closed-form anchor is not meant for deployment. In the paper it is an intervention that tells us why the trained model is inaccurate. Because it supervises exactly the quantity we evaluate, some improvement is expected, as the reviewer says. The informative results are those that the account "you supervised the metric" does not predict:

  1. The gap is not a capacity limit. The same architecture can represent a solution about 8× more accurate (0.0042 vs CE 0.033 at K=128). CE with 4× / 8× more steps gets worse at K=128 (0.052 / 0.074), drifting toward memorisation, and larger CE models also get worse there (d=256: 0.093; 4 blocks: 0.087). The residual gap is an effect of optimisation and generalisation under finite task diversity.
  2. A weak anchor does nothing, and the anchor must remain active. With a per-token weight of 0.05 the anchor is indistinguishable from CE (0.030). For the anchor-only model (0.0081 / 0.0067 at K=128 / 2048), switching the anchor off at mid-training loses most of the gain (0.028 / 0.014), and switching it on only at mid-training recovers most of it (0.0056 / 0.0078). The anchor does not "unlock" a self-sustaining circuit early in training; it keeps CE from drifting toward the pool-specific solution.
  3. No closed form is needed. A learned teacher works as the target: a CE-only model trained on unlimited synthetic tasks (PFN-style) that never sees the posterior. It gives 0.0117 / 0.0100 / 0.0090 at K=32 / 128 / 2048, versus CE 0.253 / 0.033 / 0.018. A posterior under a mis-specified prior (Dirichlet α=0.3 or 3) is no better than CE at K=128 (0.034–0.045 vs 0.033) and worse at K=2048 (0.034–0.039 vs 0.018), and an EMA self-teacher collapses to the unigram predictor. The target must generalise better than the student; it need not be exact.
  4. The target must match the task's order. For second-order chains (V=4, K=128), the trigram-posterior anchor alone gives 0.011 (0.0034 with the full recipe; CE: 0.050±0.072), whereas the order-mismatched bigram-posterior anchor gives 0.116 and collapses to the unigram predictor.
  5. The effect transfers. The same pattern appears when fine-tuning a pretrained LLM (Qwen2.5-0.5B, K=128: CE drifts to memorisation at 0.098; the exact anchor gives 0.0005; the learned-teacher anchor gives 0.0065). It also appears in PFN-style Bayesian linear regression (K=128: NLL 0.074; exact anchor 0.024; learned teacher 0.024). Details are in our response to Reviewer dSkL.

"Why not impose the attention pattern directly?" We ran this: layer-1 attention hard-coded to the previous token (φ=1.00).

KCEfixed attn + CEfixed attn + anchorSAIP
320.2530.1850.00790.0055
1280.0330.0320.00680.0042
20480.0180.057±0.0650.00650.0040
  • Imposing the circuit does not improve accuracy once CE generalises (K≥128). At K=32 it only partly reduces memorisation (0.185, still 2-Mem nearest). At K=2048, 2 of 10 seeds collapse to the unigram predictor. Combined with the anchor, fixed attention is worse than learned attention with PTA.
  • PTA alone behaves similarly (φ≈0.98–0.99): 0.032 at K=128, with 4 of 10 seeds collapsing at K=2048.
  • A correct previous-token circuit is therefore neither necessary for accuracy (the anchor alone reaches 0.0081 with φ≈0.52) nor sufficient. This dissociation is our evidence for the "strategy / circuit / accuracy" separation the reviewer questions below, and we will move it into the main text.

SBI. We agree that SBI adds nothing measurable. With SBI removed, results are 0.0056 / 0.0045 / 0.0042 at K=32 / 128 / 2048, versus SAIP 0.0055 / 0.0042 / 0.0040; the same holds at widths 128 and 256. We will drop SBI from the proposed recipe and keep it only as an ablation. PTA gives a small, consistent gain on top of the anchor (without PTA: 0.0084 / 0.0053 / 0.0052).

Part 2/3 — Definitions, evaluation protocol, Posterior Distillation, Figure 2(b)

Notation. πθ(⋅∣x1:t)\pi^\theta(\cdot\mid x_{1:t}) is the model's next-token distribution, i.e. the softmax of its logits at position tt. Every divergence is KL(π^∙ ∥ πθ)\mathrm{KL}(\hat\pi^{\bullet}\,\|\,\pi^\theta), with the reference first. The two memorising predictors are posterior mixtures over the training pool {Pk}k=1K\{P_k\}_{k=1}^K with a uniform prior over kk:

  • 2-Mem: ∑kwk(x1:t) Pk(xt,⋅)\sum_k w_k(x_{1:t})\,P_k(x_t,\cdot), with wk∝πk(x1)∏s<tPk(xs,xs+1)w_k\propto \pi_k(x_1)\prod_{s<t}P_k(x_s,x_{s+1}).
  • 1-Mem: the same mixture with unigram-likelihood weights ∏s≤tπk(xs)\prod_{s\le t}\pi_k(x_s), as in Gibson et al. (2026).

We will define all four predictors, with their likelihoods, in a self-contained table.

Test sequences. The reviewer is right that this was not stated, and in fact the original evaluation resampled sequences from the training pool. We now evaluate on both the training pool and fresh transition matrices drawn from the prior.

  • SAIP is nearly identical on the two (K=32: 0.0043 pool vs 0.0055 fresh; K≥128: within 0.0003).
  • CE differs strongly at small K (K=32: 0.142 vs 0.253), as memorisation would predict.

The revision reports both.

K1∗K_1^* and Figure 2(b). In the submission, the dashed line in Fig. 2(b) is Gibson et al.'s reference value (≈94), not our measured boundary. That is why no visible transition coincides with it. We now classify phases as Gibson et al. do: along training, the nearest of the four predictors by KL averaged over positions, computed on half pool and half fresh sequences. With 5 seeds:

  • CE is 1-Mem at K=32 (5/5 seeds).
  • CE at K=64 reaches 2-Gen in 4/5 seeds, late in training (steps 4000–6250).
  • CE at K≥96 reaches 2-Gen at steps 2000–4250.
  • SAIP is 2-Gen at every K from the first evaluation (step 500).

Hence the CE boundary is K1∗∈(32,96]K_1^*\in(32,96] with K=64 borderline, and SAIP's boundary is ≤32\le 32. We will redraw Fig. 2(b) with the measured classification per (K, step). We will define K1∗K_1^* as the smallest tested K whose final classification is 2-Gen, and note that it is resolved only at the tested pool sizes.

Posterior Distillation. This is our own baseline. The model is trained only on KL(π^2-Gen ∥ πθ)\mathrm{KL}(\hat\pi^{2\text{-Gen}}\,\|\,\pi^\theta) at all positions, without the CE term, using the same architecture and optimiser. Hahn & Goyal (2023) develop a theory of in-context learning as implicit structure induction and do not propose this procedure; citing them as its source was a mistake, which we will correct. The objective is standard output-level knowledge distillation (Hinton et al., 2015). In the corrected re-implementation it gives 0.0067 / 0.0064 / 0.0053 at K=32 / 128 / 2048 (SAIP: 0.0055 / 0.0042 / 0.0040). Every other baseline will also be given its one-line objective in a table.

Part 3/3 — Diagnostics, broad statements, and corrections

Around line 358: interpolation and "previous-token identity".

  • Previous-token identity is the value of xt−1x_{t-1}. The diagnostic groups layer-1 residual vectors at position tt by xt−1x_{t-1} and reports between-class variance over total variance, i.e. how much of the representation is explained by which token came before.
  • Linear interpolation asks a narrower question: is there a low-error straight path between two trained solutions? It does not establish separate basins.

We agree these diagnostics were poorly motivated, and that "Stronger organization … accompanies improved final prediction" is not supported as a mechanistic claim. In the revision we remove that sentence and move the interpolation and residual-variance probes to the appendix as descriptive diagnostics. The main text keeps one causal chain:

  1. strategy (phase classification);
  2. circuit (φ, plus the fixed-attention intervention);
  3. accuracy (KL);
  4. where the gap lies (the submission's logit lens shows the methods agree after layer 1 and split after layer 2, consistent with the fixed-attention result).

"Separation between strategy selection, estimator accuracy and circuit structure." We will ground this sentence directly in three controlled results:

  • CE at K≥96 is classified as 2-Gen, i.e. it has the correct strategy, yet it is 4–10× less accurate than SAIP;
  • hard-coded previous-token attention (φ=1) gives no gain in accuracy;
  • the anchor without any attention intervention leaves φ≈0.5, yet is 4× more accurate than CE.

Corrections found during re-implementation. All numbers in this response use the corrected code:

  1. the evaluation's 2-Gen reference omitted the latest transition (fixed);
  2. the Mem references indexed the true task (now posterior mixtures, as defined);
  3. evaluation used the training pool only (now pool and fresh);
  4. the anchor KL was summed over the 128 positions, so λ=0.05 is a per-token weight of 6.4; the abstract's "lightly regularizes" is inaccurate and will be corrected;
  5. the "no-SBI" ablations kept a trainable bias initialised at 0 (the bias is now removed);
  6. training and evaluation now use separate random streams.

After these corrections, SAIP reduces KL to the 2-Gen predictor by 92.7% on fresh matrices (ratio of means over 9 pool sizes × 5 seeds; equal-weight mean 85.1%; minimum 70.4%). Sequences start from the stationary distribution of their own matrix. Under this initialisation the count predictor is within KL 0.0002 of the exact Bayes predictive, and results are unchanged with uniform initialisation (details in our response to X7n7).

Anchor weight. The main setting is λ=0.05, while some appendix ablations used λ=0.02; we will label these clearly. A new sweep (3 seeds, K=128; the λ=0.05 value is from the main 5-seed run) gives SAIP 0.018 / 0.012 / 0.0065 / 0.0042 / 0.0039 / 0.0037 at λ = 0.005 / 0.01 / 0.02 / 0.05 / 0.1 / 0.2, saturating beyond 0.05.

We hope these answers and the new controls address the reviewer's concerns, and we would welcome further questions.

Reviewer_X7n7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors study the problem of in-context prediction of Markov data via two-layer transformers. The problem is popular in existing literature; the authors mostly draw from Gibson et al. (2026) and study the phase transition between memorisation of the data distributions and generalization to count-based in-context estimators as a function of the number of distributions in the training pool. The objective of the authors is to study whether this threshold (and the overall cross-entropy loss) changes when different architectural and optimization changes are applied to the transformer architecture. The ensemble of these modifications is termed Symmetry-Aware In-Context Pre-training (SAIP) in the paper. Overall, the experimental results point to a strong reduction in KL divergence between the output of the model and the count-based bigram estimator.

Critical strengths

The paper's strength is in the plethora of well-designed experiments and visualizations.

Critical weaknesses

The main weakness of the paper is the lack of a clear objective or take-away point that is meaningful outside of the small toy data model studied in the paper.

As a researcher that worked on the topic myself, I believe that the value of the Markov model was to understand how and how well LLMs could naturally learn to implement provably optimal estimators in practice, hoping to understand the role of each component of these architectures, and whether these patterns would translate in more complex scenarios like language modeling.

In this paper, the authors simply try to implement architectural and training modifications in order to steer the model closer to the optimal estimator that is specific and exclusive to the Markov setting. This predictor has generally no intrinsic optimality when the data setting is more complex or general, therefore the changes introduced in this paper carry little value when we move from the toy Markov setting to real scenarios.

Even if the changes proposed by the authors could improve training or inference in real models, there is no proof or experiment in the paper that investigates this possibility.

Questions for authors

My main question is: what is the take-away from this paper for the general community of machine learning researchers? Does any of the findings in this paper tell us something that goes beyond the toy Markov setting studied in the paper?

A second small technical question: how are the training sequences initialized? The Markov distribution only determines the transition probabilities. Is the initial distribution of each sequence fixed? Is it uniform? Is it the stationary distribution of the Markov kernel? This may be important because different initial distributions lead to different optimal in-context estimators.

8 comments

Part 1/3 — Main question: what is the take-away beyond the Markov setting?

We thank the reviewer for a careful and expert review. We agree that a recipe that only steers a model toward an estimator specific to Markov chains would have limited value. We therefore re-ran the full study with a clean re-implementation (5 seeds per cell; evaluation on both the training pool and fresh transition matrices) and added experiments outside the original setting. Below we first state the findings we believe transfer, then the evidence. Unless stated otherwise, numbers are KL(2-Gen ‖ model) at the final position on fresh transition matrices, averaged over 5 seeds.

Finding 1 — The residual gap comes from optimisation under finite task diversity, not from limited capacity.

  • The same two-block model can represent a solution about 8× more accurate than the one cross-entropy (CE) training finds: at K=128, SAIP reaches 0.0042 and CE 0.033.
  • More compute does not close the gap for a finite pool. With 4× / 8× more steps, CE gets worse at K=128 (0.052 / 0.074), because it drifts toward memorising the pool. It improves only when task diversity is large: 0.0074 / 0.0060 at K=2048, and 0.0067 / 0.0043 with unlimited tasks.
  • Larger models do not help either (3 seeds). At K=128, CE gives 0.051 (d=128), 0.093 (d=256) and 0.087 (4 blocks); SAIP gives 0.0039, 0.0033 and 0.0024.

Finding 2 — Strategy, attention circuit and accuracy dissociate.

  • (i) CE at K≥96 is classified as 2-Gen, i.e. it uses the correct strategy, yet it remains 4–10× less accurate than SAIP.
  • (ii) Hard-coding exact previous-token attention (φ=1.00) does not improve accuracy: 0.032 vs CE 0.033 at K=128. It is also unstable at large K: at K=2048, 2 of 10 seeds collapse to the unigram predictor.
  • (iii) Conversely, the posterior anchor alone leaves φ≈0.5, as in CE, but is 4× more accurate (0.0081).

The gap therefore lies downstream of identifying the previous token, which is consistent with the logit-lens probe in the submission: the methods agree in the read-out after layer 1 and split only after layer 2.

Finding 3 — Output-level posterior supervision does not require a closed form. We replaced the exact posterior by a learned teacher: a CE-only model trained on unlimited synthetic tasks, i.e. a PFN-style model that never sees the closed form. It gives 0.0117 / 0.0100 / 0.0090 at K=32 / 128 / 2048, versus CE 0.253 / 0.033 / 0.018. This is the practical form of the recipe: distil a generalist model trained on the full prior into a learner trained on a limited task pool.

Finding 4 — The same phenomena appear in pretrained LLMs and in a PFN regression setting (Part 2).

We will rewrite the introduction and conclusion around these findings, and present the Markov estimator as a measurement device rather than as the object to be deployed.

Part 2/3 — New evidence outside the two-block Markov setting

(a) Frozen pretrained LLMs, purely in context. We give Qwen2.5-0.5B / 1.5B / 7B Markov sequences (V=10, T=128) as comma-separated digits. We read the next-symbol distribution renormalised over the ten digit tokens, which carry >99% of the probability mass.

  • At t=128, KL(2-Gen ‖ LLM) is 0.138 / 0.117 / 0.184, far larger than for the trained two-block models (CE 0.018–0.033, SAIP 0.004).
  • All three LLMs are closer to the in-context unigram predictor, 1-Gen (0.040 / 0.051 / 0.066).
  • KL to 2-Gen is smallest around t≈32 (0.085 / 0.074 / 0.083) and grows with longer context.
  • Against the true matrix, KL(P ‖ LLM) is 0.29–0.36, while the Bayes predictor achieves 0.17.

Pretrained LLMs therefore also adopt an in-context strategy without executing the Bayes-optimal one accurately, and scaling from 0.5B to 7B does not remove the gap.

(b) A pretrained LLM fine-tuned on a finite task pool. We fully fine-tuned Qwen2.5-0.5B (2000 steps, 3 seeds) on the same Markov task pools.

KCECE + 2-Gen anchorCE + learned-teacher anchor
320.250 (2-Mem nearest)0.00230.0092
1280.098 (2-Mem nearest)0.00050.0065
20480.00110.00010.0056
∞0.00070.00010.0054
  • At K=128, CE first approaches the generalising solution (0.023 at step 500) and then drifts to memorisation (0.098 at step 2000). This is the transient pattern; both anchors prevent it.
  • Qwen2.5-1.5B (K=128, 3 seeds) shows the same pattern: CE drifts to memorisation (0.102, 2-Mem nearest), while the exact anchor gives 0.0005 and the learned-teacher anchor 0.0066.
  • With abundant tasks, the pretrained LLM is already accurate under CE. For LLMs the auxiliary target therefore acts mainly on generalising from limited task diversity.
  • The teacher-anchored LLM tracks the accuracy of its small teacher (teacher 0.0053–0.0123 across seeds; anchored LLM 0.0046–0.0097).

(c) PFN-style Bayesian linear regression. Setup: D=8, 32 in-context pairs, Gaussian noise, a 4-layer transformer with a Gaussian output head, 5 seeds. The metric is KL(posterior predictive ‖ model) on fresh tasks.

KNLLNLL + exact PPD anchorNLL + learned-teacher anchor
323.48 (dMMSE nearest)0.270.23
1280.0740.0240.024
5120.0240.0170.017
20480.0200.0150.016
∞0.0190.0140.015
  • The teacher is trained with NLL only, on unlimited tasks, and never uses the closed form. The anchor helps at every pool size, and the learned teacher matches the exact target.
  • At K=32 the anchor changes the strategy, from memorising the pool to ridge-like behaviour, but the error on fresh tasks remains large.

Limitations we will state. In all three settings the gain comes from output-level supervision by a predictor that generalises better than the student. We do not claim that SBI or the attention prior transfer, and we do not claim gains for language-model pre-training in general.

Part 3/3 — Initial distribution, and corrections found during re-implementation

How are training sequences initialised? Thank you for this question; the paper did not state it. In our code, x1x_1 is drawn from the stationary distribution πk\pi_k of the sequence's own matrix PkP_k. As the reviewer notes, the Bayes predictor then differs from the Dirichlet count estimator, because x1x_1 carries information about PP. We quantified the difference in three ways.

  • Exact posterior predictive. We computed it under stationary initialisation by importance sampling: 2048 Dirichlet-posterior samples per sequence, weighted by πP(x1)\pi_P(x_1), validated against 2-D quadrature for V=2. On 512 fresh sequences, KL(exact ‖ 2-Gen counts) = 0.00018. This is over 20× smaller than SAIP's residual: KL(exact ‖ SAIP) = 0.0044 and KL(exact ‖ CE) = 0.033 at K=128.
  • Uniform-initialisation control. We retrained all methods with uniform initialisation, under which the count estimator is exactly Bayes-optimal. CE / SAIP give 0.033 / 0.0044 at K=128 and 0.018 / 0.0042 at K=2048, unchanged from stationary initialisation.
  • Mem predictors. Their likelihoods now include x1x_1 under each PkP_k, as in Gibson et al. (2026).

The revision will state the initialisation, give the exact predictive, and report the uniform-initialisation control.

Corrections. While re-implementing the pipeline we found and fixed the following. All numbers in this response use the corrected code.

  1. The evaluation's 2-Gen reference omitted the most recent transition.
  2. The 1-Mem / 2-Mem references indexed the true task. They are now posterior mixtures over the pool, as the paper defines them.
  3. The original evaluation resampled sequences from the training pool only. We now report both the pool and fresh matrices.
  4. The anchor KL was summed over the 128 positions, so λ=0.05 is a per-token weight of 6.4 relative to CE. "Lightly regularizes" was therefore inaccurate. With a per-token weight of 0.05 the anchor has no measurable effect: 0.030 vs CE 0.033 at K=128. We will state the normalisation explicitly.
  5. The "without SBI" ablations initialised the bias at 0 but kept it trainable. The bias is now removed in those ablations.

Size of the improvement. With these corrections, SAIP reduces KL to the 2-Gen predictor by 92.7% on fresh matrices (ratio of means over 9 pool sizes × 5 seeds; equal-weight mean 85.1%; minimum 70.4%) and by 90.4% on the training pool. This is consistent with the 89.3% in the submission. We stress that this is a reduction in divergence from the 2-Gen posterior — the quantity the anchor supervises — not Bayes optimality. Measured against the true transition matrix, which the anchor never sees, the excess risk over the Bayes predictor falls from 0.032 to 0.0041 nats at K=128.

We hope these additions address the reviewer's concerns about scope and take-away, and we would be grateful to hear whether the reframing above is convincing.

Comments on the paper

1 comment

Dear reviewers,

This is a friendly reminder to read the authors’ rebuttals and post your responses by Nov 18. Please briefly indicate which concerns have been addressed and which remain unresolved. Even if your overall assessment remains unchanged, please provide a brief response to the authors. Please also update your reviews and scores as appropriate. Thank you for your time and effort!

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.