- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper aims to show, through a controlled experiment, that a transformer can adopt the right inference strategy for a given task through ICL without necessarily making accurate predictions. Moreover, it suggests that training can improve accuracy when guided by an appropriate auxiliary task loss. Following the Markov-ICL benchmark and two-block transformer of Gibson et al. (2026), the paper uses simple Markov sequences, four Bayesian predictors covering memorization and generalization, and a simplified two-layer transformer. What is added to Gibson et al. is a training recipe called SAIP; small attention modification, an attention auxiliary loss, and a posterior-matching loss that directly supervises the generalizing Bayesian predictor. The experiments show that SAIP drives generalization at smaller task-pool sizes and improves accuracy, with most of the accuracy gain coming from the posterior-matching loss.
Critical strengths
- The paper contains ample simulation and ablation results showing SAIP’s ability to improve accuracy on the Gibson Markov-ICL benchmark.
- The improvement from the posterior loss after the phase transition is illustrative. It may help us think about how auxiliary losses, commonly used in deep learning methods such as REPA, improve performance.
Critical weaknesses
-
The biggest weakness is its scope. The idea that transformer ICL may adopt the right inference strategy without implementing it accurately seems plausibly relevant to ICL in pretrained LLMs. However, the current experiments are so limited that the conclusion is largely confined to improving accuracy on the Gibson Markov-ICL benchmark. What can be established so far is that, at least for this specific benchmark task, auxiliary posterior supervision helps. Extending this conclusion would require additional experiments in more general settings, including open-source LLMs. Prior-fitted network (PFN) pretrained with synthetic task could also be informative.
-
Second, the proposed SBI attention modification has an intuitive motivation, but the ablations show that most of the accuracy improvement comes from the posterior loss. This leaves limited evidence that SBI itself provides a meaningful additional benefit to prediction accuracy.
-
Finally, as the authors acknowledge, the posterior loss requires a known, tractable posterior. This is available in the synthetic benchmark, but its applicability to more realistic tasks remains unclear.
Questions for authors
Q1. I wonder whether “learning an inference strategy but implementing it inaccurately” is representative of real-world tasks or mainly arises in Gibson et al.’s setting. It is often argued that neural networks generalize by learning algorithms shared across tasks [1, 2]. Similarly, the literature on ICL [3, 4] studies how transformers can implement underlying inference algorithms, such as gradient descent and Bayesian prediction. How do the current findings relate to this broader view, and what evidence suggests that the observed gap extends beyond this benchmark?
[1] Xu, K., Li, J., Zhang, M., Du, S. S., Kawarabayashi, K.-i., and Jegelka, S. (2020). What Can Neural Networks Reason About? ICLR
[2] Veličković, P., and Blundell, C. (2021). Neural Algorithmic Reasoning. Patterns
[3] Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. (2023). Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning. NeurIPS
[4] Kang, G., Lee, C. J., and Cheng, X. (2026). Transformers Can Learn Posterior Predictive Distributions In-Context. ICML
Q2. Relatedly, can the authors explain how the current results extend to more general settings, such as next-token prediction in LLM pretraining or PFN for tabular prediction, and provide evidence of their relevance? If so, what implications do these results have for pretraining or posttraining, architectural modifications, or controlled interventions?
11 comments
Part 2/3 — Q1 (representativeness and the algorithmic view) and Q2 (implications)
Q1. Is "right strategy, inaccurate execution" specific to Gibson et al.'s setting? The new evidence suggests it is not:
- the gap appears in frozen pretrained LLMs and is not removed by scale (Part 1a);
- when an LLM is fine-tuned on a finite task pool, CE first reaches and then leaves the generalising solution (1b);
- in regression, an NLL-trained PFN stays measurably away from the posterior predictive (1c).
Relation to the algorithmic view.
- Existence vs. what training finds. Ahn et al. (2023) and Kang, Lee & Cheng (2026) show that transformers can implement, in context, preconditioned gradient descent or the posterior predictive of GP regression. These results characterise what a transformer implements at an optimum, a critical point or an explicit construction, typically with unlimited task samples. Our results concern whether standard training finds an accurate implementation when task diversity is finite. In our setting it does not, although an accurate solution exists within the same architecture (0.0042 vs 0.033 at K=128).
- Algorithmic alignment. Xu et al. (2020) relate sample efficiency to how well an architecture's structure matches the algorithm. The two-block architecture matches the count-and-normalise algorithm well enough to represent it accurately, yet from a finite pool CE selects a less accurate solution. Alignment governs what is learnable; it does not by itself determine which of several fitting solutions training selects.
- Supervising the computation. Neural algorithmic reasoning (Veličković & Blundell, 2021) often supervises intermediate algorithmic steps. Our anchor is an output-level analogue. Notably, directly supervising the intermediate circuit does not improve accuracy: hard-coded previous-token attention (φ=1) gives 0.032 vs CE 0.033 at K=128 and is unstable at K=2048. What helps is a target that generalises better than the student, whether exact or learned.
Q2. Implications, stated as testable claims.
- Pre-training on synthetic priors (PFNs). When a PFN is trained on a finite set of prior samples, or fine-tuned on a narrower family, an auxiliary distillation target from a generalist trained on the full prior mitigates drift toward the finite pool. This holds in our Markov, regression and LLM fine-tuning results, at the cost of one teacher forward pass per batch.
- Post-training / fine-tuning on narrow task families. The pattern of transient generalisation followed by memorisation appears when fine-tuning a pretrained LLM (1b). Monitoring, and anchoring to, the divergence from a generalising reference is a cheap control.
- Architectural or attention interventions. Imposing the "correct" attention pattern did not improve accuracy in any setting we tried, and the SBI bias contributes nothing measurable (see below). We therefore do not recommend architectural priors as the lever, and we will report this as a negative result.
- Controlled interventions. The anchor must stay active. Removing it at mid-training loses most of the gain (K=128: SAIP 0.0042 → 0.022; anchor alone 0.0081 → 0.028), while switching it on only at mid-training recovers it (0.0056). It acts as a continual regulariser, not as an early "circuit unlock".
Scope of these claims. We did not run LLM pre-training or tabular PFNs (e.g. TabPFN); Bayesian linear regression is the canonical PFN prior we could test within the rebuttal period. The claims above therefore concern distillation under limited task diversity, not pre-training at scale.
SBI. We agree that SBI gives no meaningful benefit. With SBI removed entirely (absent, not merely zero-initialised), results are 0.0056 / 0.0045 / 0.0042 at K=32 / 128 / 2048, versus SAIP 0.0055 / 0.0042 / 0.0040; the same holds at widths 128 and 256. We will remove SBI from the proposed recipe and present the method as CE + posterior anchor, plus a small PTA gain (0.0053 → 0.0042 at K=128).
A known, tractable posterior. Our main addition is the learned-teacher anchor: a CE-only model trained on unlimited synthetic tasks, with no closed form anywhere in the pipeline.
- In the Markov setting it gives 0.0117 / 0.0100 / 0.0090 at K=32 / 128 / 2048, versus CE 0.253 / 0.033 / 0.018.
- In PFN regression it matches the exact target.
- The teacher must generalise better than the student but need not be exact. A mis-specified prior (Dirichlet α=0.3 or 3) is no better than CE at K=128 and worse at K=2048, and an EMA self-teacher collapses.
Part 3/3 — Corrections and revision plan
While re-implementing the pipeline for the rebuttal we fixed several issues. All numbers in this response use the corrected code:
- the evaluation's 2-Gen reference omitted the latest transition;
- the Mem references indexed the true task instead of using the posterior mixture;
- evaluation used only the training pool (we now also report fresh matrices);
- the anchor KL was summed over the 128 positions, so λ=0.05 is a per-token weight of 6.4, not a "light" regulariser;
- the ablations "without SBI" kept a trainable bias.
Size of the improvement. With 5 seeds and 9 pool sizes, SAIP reduces KL to the 2-Gen predictor by 92.7% on fresh matrices (equal-weight mean 85.1%; minimum 70.4%), consistent with the submitted 89.3%. Against the true transition matrix, which the anchor never sees, the excess risk over the Bayes predictor falls from 0.032 to 0.0041 nats at K=128. With stationary initialisation the count predictor is within KL 0.0002 of the exact Bayes predictive, and a uniform-initialisation control gives the same results.
Further robustness.
- Model size (3 seeds): at K=128, CE / SAIP give 0.051 / 0.0039 at width 128, 0.093 / 0.0033 at width 256, and 0.087 / 0.0024 with 4 blocks.
- Second-order chains (K=128): the full recipe with a trigram anchor gives 0.0034 (anchor alone 0.011) versus 0.050 for CE; a bigram anchor fails (0.116).
- LLM input format: without comma separators the frozen LLMs show the same picture (KL(2-Gen‖LLM) 0.15–0.19 at t=128; digit-token mass 0.94–0.96).
- Anchor weight (3 seeds, K=128): SAIP improves monotonically over λ ∈ [0.005, 0.2] and saturates beyond 0.05 (0.018 → 0.0042 → 0.0037).
Revision plan.
- Reframe the paper around the transferable findings: the gap under finite task diversity, the dissociation of strategy, circuit and accuracy, and anchoring to a generalist teacher.
- Add the LLM and PFN experiments as a new section.
- Present the learned-teacher anchor as the practical, non-oracle version of the method.
- Drop SBI from the recipe.
- Report all corrections and the uniform-initialisation control.
We hope these experiments address the reviewer's questions on scope and relevance, and we would be glad to discuss further.
Part 1/3 — New experiments: open-source LLMs and a PFN setting
We thank the reviewer for a constructive review and for the concrete suggestion to test open-source LLMs and PFNs; we ran both. Unless stated otherwise, numbers are means over seeds, evaluated on fresh tasks drawn from the prior rather than on the training pool, from a clean re-implementation (see Part 3).
(a) Frozen pretrained LLMs, purely in context. Qwen2.5-0.5B / 1.5B / 7B read Markov sequences as comma-separated digits (V=10, T=128, Dirichlet(1) rows, fresh matrices; 1024 sequences, 2 seeds). We use the next-symbol distribution over the ten digit tokens, which carry >99% of the probability mass.
This is the "strategy learned but executed inaccurately" phenomenon, now in pretrained LLMs.
(b) Fine-tuning a pretrained LLM on a finite task pool. Full fine-tuning of Qwen2.5-0.5B (2000 steps, batch 32, lr 3e-5, 3 seeds):
(c) PFN-style in-context Bayesian linear regression, the setting of Raventós et al. (2023). Tasks are w∼N(0,I8) drawn from a finite pool of K tasks, with 32 noisy (x, y) pairs per sequence. The model is a 4-layer transformer with a Gaussian output head trained by NLL, run with 5 seeds. The metric is KL(posterior predictive ‖ model).