acceptodds
Under review as a conference paper at ICLR 2027

Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study

Abstract

In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice? To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: sufficient-statistic invariance and the martingale property. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.

Primary area:
reinforcement learning
Keywords:
In-Context Reinforcement Learning, Bayesian Posterior Sampling, Decision-Pretrained Transformer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZTRdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study investigates whether in-context reinforcement learning behaves as Bayesian posterior sampling. Using decision-pretrained transformers, it tests two necessary conditions, invariance to sufficient statistics and the martingale property, for Bayesian decision-making. In experiments across Gaussian, Bernoulli, and linear bandit problems, the former condition is violated in all models, while the latter is largely violated in the variant models as well.

Critical strengths

While many studies have suggested that ICL or ICLR is Bayesian in some sense, I believe that investigating what inference algorithms ICRL actually implements is significant.

Critical weaknesses
  1. The overall framework of using a martingale test to demonstrate that ICL is non-Bayesian overlaps with Falck et al. So, I consider this to be an incremental contribution.

  2. A more importantly, I think the GPT-2 used in this work is too small. For instance, studies on Bayesian ICL, such as Ma et al., Wakayama et al., and Xie et al., use significantly larger GPT-2 architectures and much larger pretraining datasets. Consequently, when this study claims that DPT is non-Bayesian, I suspect the underlying cause stems from incomplete pre-training (e.g., insufficient optimization, inadequate sample size, or limited model capacity) rather than an intrinsic limitation of the ICRL mechanism.


Falck et al. (2024). Is In-Context Learning in Large Language Models Bayesian? ICML2024.

Ma et al. (2026). Optimal In-context Adaptivity and Distributional Robustness of Transformers. arXiv:2510.23254.

Wakayama et al. (2025). In-Context Learning Is Provably Bayesian Inference. ICML 2026.

Xie et al. (2022). An Explanation of In-context Learning as Implicit Bayesian Inference. ICLR 2022.

Questions for authors

The first contribution bullet states that "We...derive two necessary conditions from it, namely sufficient-statistic invariance and the martingale property". However, aren't these conditions already well-established in the prior work? Specifically, does the former appear in Appendix C of Wakayama et al. as well as in classical Bayesian statistics literature? And is the latter a restatement of the arguments in Falck et al.?

If these do indeed represent distinct and novel contributions, please clarify your specific claims so that I will reconsider the informativeness score. Otherwise, explicitly clarifying that these are existing properties while respectfully referencing prior work would improve the soundness of the paper.

Reviewer_uMu6OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper concerns in-context reinforcement learning (ICLR) for multi-armed bandits. Specifically, it empirically tests whether ICLR is a Bayesian posterior sampler in practice, which is only established under strong assumptions. In order to be able to test this, necessary conditions are established and two variations of decision-pretrained transformers are proposed. The experimental results show that the conditions are violated in Gaussian, Bernoulli and linear bandits settings.

Critical strengths

The paper is a valuable integration of theoretical and experimental contributions. It is important that theoretical results are tested in practice.

The paper is very clearly written, has a clear narrative and is strongly executed.

The paper provides an approach to test Bayesian claims about ICRL (some more discussion about the value of the techniques outside the specific setting would be appreciated).

Critical weaknesses

While I appreciate that negative results are definitely of value, the paper would be stronger if the authors could also dig into what the main reason of the violations is (i.e., is it the lack of expressivity of the model or finite training data). The ablation study in Sec 4.3 is a good first step in that direction, however.

Questions for authors

Can the authors comment on the identified weakness? I fully realize that this is beyond the scope of the current paper, but some thoughts on this issue would be highly appreciated.

Can the authors discuss potential extensions to sequential settings?

As a suggestion, I would recommend being a bit more explicit in the abstract/introduction about the focus on ICRL in a multi-armed bandit setting.

Reviewer_PfKMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Prior work introduced the decision pretrained transformer, and proved that asymptotically and with a sufficiently expressive model, DPT implements Bayesian posterior sampling.

This paper considers whether this holds in finite data settings for specific types of domains when using generic architectures. Specifically, the authors introduce two properties that must hold for multi-armed bandit settings: sufficient-statistic invariance and the martingale property, and hypothesis tests for evaluating whether these properties hold for DPT and some variants. For Gaussian, Bernoulli and linear bandit settings, they find strong evidence these assumptions are violated, even when DPT matches the (regret) performance of thompson sampling in their experiments (e.g. in the Bernoulli and linear bandit settings). The implication is that in practice, these models are not mimicking all the properties of Bayesian posterior sampling, and this persists even when they modify the training procedure to be permutation invariant.

Critical strengths

Strengths

  • Thompson sampling is a powerful technique but often hard to apply in practice, and DPT offered a generic transformer-based way to approximate it. Therefore, understanding if DPT is not matching all properties of bayesian posterior sampling in finite data is useful.
  • The authors introduce two clear properties for multi-armed bandit posterior sampling
  • The authors give clear evidence that the existing DPT method and a couple variants they introduce to satisfy proposition 1, do not satisfy both their assumptions.
  • It was useful to know that training for permutation invariance did not resolve the martingale property.
Critical weaknesses

Weaknesses

  • The results are only for k-armed bandits, with no context and no sequential decision making across a horizon. The title refers to reinforcement learning, but the focus is much more narrow, including that their two conditions are specific to this setting— aka these conditions would need to be extended to account for context and sequences of decisions.
  • In some ways it is not so surprizing that these assumptions are violated, since DPT does not directly train for them (no permutation invariance, nor the martingale property).
  • It is mostly left open how to address these limitations, and perhaps especially important, whether they matter in terms of performance or other questions we might care about. The permutation ablation is useful, since it’s a simple way to correct this. But given that performance on both the linear and bernoulli bandit matches posterior sampling (similar to results found in DPT), there may be a question of finding situations or questions where this significantly hurts performance.
Questions for authors

In addition to hearing feedback about the above weaknesses, it was surprizing to me that the Gaussian DPT did not match PS but DPT did match PS for linear an bernoulli. Could the authors expand on this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.