acceptodds
Under review as a conference paper at ICLR 2027

Transformers develop separable posterior and value representations in a simple decision task

Abstract

How does a transformer make decisions when it is uncertain about the world? A network could form a posterior over hidden states and value its options under that posterior, or it could map observations to choices with no such intermediate structure. We investigate this question with a transformer trained by reward alone on a simple decision task. Hidden states are assigned payoffs, and offers composed of sets of hidden states are presented to the transformer. The offers and state payoffs are given in context, and the exact posterior and the value of every offer are known in closed form. We anticipate and then find that both quantities are linearly represented in the residual stream (), in separate subspaces, and that the representations extend to offers never seen in training. Interventions show that both are used for choice: steering the posterior shifts choice by the amount the ground truth predicts, and does so through the value subspace, while steering the value shifts choice and leaves the posterior fixed. None of this is required by the objective, which constrains only the difference between the offer values. A pretrained language model, Qwen3-1.7B-Base, trained on the same task, develops the same organization. Our work extends the posterior geometry found under next-token prediction to choice, and gives a setting in which the quantities that drive a model's decisions can be measured against ground truth and intervened on.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, linear probes, activation steering, causal intervention, emergent representations, world models, belief states, reinforcement learning, decision theory, toy models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pzUiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether transformers trained on a synthetic decision task form separable representations of posterior beliefs and action values. In the task, exact posteriors and offer values are analytically computable. The authors train a small transformer with reward-only REINFORCE and show that both quantities are linearly decodable, occupy largely distinct subspaces, generalize to held-out offers, and causally affect choices under activation steering. They replicate the main findings in Qwen3-1.7B-Base trained on the same symbolic task. The paper relates these results to decision-theoretic notions of belief and desire.

Critical strengths
  • Clean ground-truth setup. The synthetic task makes both posterior and offer value exactly computable, enabling unusually controlled probing and intervention analyses.

  • Causal evidence beyond probing. The paper includes steering and mediation-style insertion/removal experiments, which make the evidence stronger than linear decodability alone.

  • Interesting link between posterior geometry and choice. Extending prior posterior-geometry results from prediction to a decision setting is a natural and useful direction.

Critical weaknesses
  • The claims are stronger than the setting supports. The results are framed as evidence for transformer posterior/value representations relevant to decision making, but all main evidence comes from a highly engineered toy task where posterior-to-value computation is the natural solution.

  • Limited novelty over prior work. The posterior geometry largely follows prior HMM work, and the probing/steering tools are standard. The main contribution is their combination in a new synthetic task, which feels incremental.

  • Weak external validity. The Qwen3 experiment uses the same symbolic toy task, not natural-language decisions or realistic preference/reward settings. It shows scale replication, but not generality to real LLM behavior.

  • The reward-only framing is overstated. Payoffs and offer masks are explicitly given in context, and each decision is a contextual bandit. The task avoids long-horizon credit assignment and does not show that utilities are learned from rich interaction.

  • Separable-subspace evidence depends on probe construction. The claimed separation relies on ground-truth-fitted linear probes and covariance-based subspaces. It is unclear how robust this is to alternative probe definitions, task variants, or more distributed/nonlinear representations.

  • Held-out generalization is too small. Generalization is tested in a nine-state world with only 511 possible nonempty masks and 64 held-out masks. This is useful but too limited to support broad claims.

Questions for authors
  1. Stronger evidence is needed that the posterior/value organization appears beyond this synthetic task, ideally in more naturalistic language-based decision settings.

  2. More discussion is needed on whether the task effectively bakes in the desired posterior-to-value decomposition.

  3. Additional baselines would help show whether this organization is transformer-specific, objective-induced, or simply task-induced.

  4. Please test robustness to probe construction, subspace definitions, read/write layer choices, and nonlinear or distributed alternatives.

  5. The Qwen3 experiment should be better justified as evidence about LLMs, since it is still trained on the same symbolic toy task.

  6. Larger and harder generalization tests would strengthen the paper, e.g., larger state spaces, more factors, unseen payoff structures, or held-out structural classes.

1 comment

We thank the reviewer for their thoughtful feedback and for recognizing the value of our ground-truth setup, causal evidence, and relevance to prior literature. Below, we address their points and the corresponding revisions to the manuscript.

W1 (Claims stronger than evidence supports) We chose a toy decision task for the reason the reviewer lists as a strength: the posterior and offer value can be computed in closed form. Our claims are in regard to this setting, rather than a statement about how transformers behave in general. We have adjusted the claims in the revised manuscript to clarify this, and the limitations paragraph of the Discussion has been extended.

W2 (Limited novelty) We agree with the reviewer that the probing and steering methodology is standard and that the posterior geometry was established by Shai et al. (2024, 2026) under next-token prediction. In this work, we extend this to a setting in which the model is rewarded based on choices, with no next-token prediction, and find that the same posterior geometry forms. Furthermore, to our knowledge, no prior study has characterized posterior and value representations in a single network against ground truth. We aim to extend prior work showing how posterior geometry is represented towards understanding how these representations may be used in practice.

W3, Q1, Q5 (External validity of the Qwen3 experiment) We agree with the reviewer that the Qwen3 experiment does not reflect natural, plain-language interactions. Instead, we show that the findings are not specific to a small transformer trained from scratch: the results replicated in Qwen3, which has 28 layers and a residual width of 2048. We aim to extend these findings to more naturalistic settings in future work.

W4 (Reward-only framing overstated) The reviewer is correct to point out that the task is a contextual bandit, and that the offers and state payoffs are given in context rather than learned from reward. By "reward alone", we meant that the only training signal the model gets is from the sampled reward, and neither the posterior nor the value is directly supervised. To make this clear, we have revised the manuscript to say "choice reward". We leave settings with long-horizon credit assignment to future work.

W5, Q4 (probe construction, subspace definitions, and layer choices) We agree with the reviewer and have added new analyses to the revised manuscript. Using one and two layer MLP probes, we find no evidence that additional information is represented nonlinearly (Appendix D). We now show that the joint posterior is decoded beyond what can be explained by the factor posteriors (R² of 0.894 against 0.761; Section 3.2). However, using a causal ablation intervention, we found that the joint representation is not used for making decisions (slope of 0.000; Section 3.4). Steering through principal components of the activations, rather than subspaces fit to the ground truth, gives the same result (value slope of 0.944; posterior slopes of 0.997 and 0.991; Section 3.4). Appendix E shows the probing and steering results across layers. The linear probe accuracy changes little under modifications to the probes' regularization, training-set size and readout position (Appendix D).

W6, Q6 (held-out generalizations) We agree with the reviewer that the state space is small and have added two generalization tests. First, we trained a model in which a structural class of offers was held out, the 27 product sets. The organization was unchanged, and on the 27 never-offered product sets the offer value is read at a median R² of 0.999 (Appendix J). We also tested a set of payoff cards that were never seen during training, on which the offer values are read at R² = 0.999 (Appendix G). We reserve testing larger state spaces and more factors for future work.

Q2 (Baked in structure) Following the reviewer's suggestion, we expanded Section 2.4 to discuss what is entailed by the task. The task requires a sufficient statistic of the clues, and a choice logit equal to β(Q(A) − Q(B)), but it does not require the organization we report. Q(A) and Q(B) need not be represented separately, yet Q(A) + Q(B) is read at R² = 0.999 (Section 3.2). The task also does not force separate subspaces, linear readability, or causal use of the representations, but we find these nonetheless (Sections 3.2 to 3.4).

Q3 (Additional baselines) We now train a network using only next-token prediction, without any choice reward, and find that the posterior representation forms, but the value representations do not (Section 3.6). We also find the same organization in a model trained using a non-factored nine-state Markov chain (Appendix L), in a setting where offers are given as conditions per chain (Appendix J), and in models with 2, 3 and 8 layers (Appendix J). We did not test architectures other than transformers.

Reviewer_gNhGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The task addressed in the paper is a one-step decision under uncertainty; namely, a contextual bandit with a belief, or a one-step POMDP. The belief is obtained by HMM filtering over 9 states (two independent 3-state chains) with fixed parameters. Each action is a lottery drawn from the belief restricted to a set of states, with per-state utilities given in context. Since payoffs and offers change every episode and are laid out state by state by position, retaining the belief and computing the masked, renormalized expected payoff is close to the only practical solution. The study involves a single task and a single encoding, the pretrained-model replication uses the same encoding, and the decision-theoretic interpretation claims much more than the evidence supports. Presenting the task in standard POMDP or decision-analysis terms would make both the setting and its limits clear. Relative to the large literature on what transformers represent, learn and generalize, the contribution reads as not too challenging case study.

Critical strengths
  • The question, whether decisions run through separable, causally used belief and value representations, is relevant to interpretability.
  • The ground truth is exact, and the causal analysis is careful: off-plane controls, mediation, held-out offers, seeds, and a pretrained model.
Critical weaknesses
  • The result is close to forced by the design. The claim that "none of this is required by the objective" is not compelling: generalizing to arbitrary in-context payoffs and offers requires a sufficient statistic of the clues, and the choice logit must equal beta·(Q(A) - Q(B)). The organization found is plausibly a property of a positional, state-by-state encoding of a fixed 9-state world.

  • Generality is limited. There is one task, one encoding, nine states, fixed dynamics and one-step decisions. The Qwen replication uses the same 17-symbol encoding and full fine-tuning, so it adds little. Relative to the large literature on what transformers represent, learn and generalize, the study reads as a single example.

  • The presentation obscures the task. Stated in standard POMDP or decision-analysis terms, the setting and its assumptions would be immediately clear. The reward drawn from the posterior rather than the true state, chosen so that Q is non-linear in the belief, is a bit artificial and unexplained.

  • Not sure that I find meaningful the first sentence in the abstract: "How does a transformer make decisions when it is uncertain about the world?" Probably there are many different encodings, training regimes, etc; not sure that the question in itself is too meaningful in general. Indeed, the paper addresses the question in a very restricted example.

Questions for authors
  1. Would the same organization arise under encodings that do not lay the task out state by state, for example offers given as conditions on the two chains, or payoffs not listed per state?
  2. Why is the reward drawn from the posterior conditioned on the offer rather than from the true hidden state? Do the findings hold under the standard formulation, where Q is linear in the belief?

== After rebuttal:

Thank you for the detailed rebuttal and the new experiments. They resolve my concrete concerns: the encoding, the reward, the framing and the overclaiming. They also add one informative result, the factorized posterior, and I have raised my informativeness score accordingly. The central limitation, however, stands: in a tiny, fixed, one-step world, the organization found is the natural solution, and the study remains a careful case study rather than a more general finding. I therefore keep my rating, but I would not oppose acceptance.

3 comments

We thank the reviewer for their constructive feedback, and for noting the relevance of our work to the field of interpretability as well as the value of our ground truth setup and causal controls. We respond to each point below and describe the corresponding changes to the revised manuscript.

W1 (forced by design) We agree with the reviewer that the task requires a sufficient statistic of the clues and that the choice logit must equal β(Q(A) − Q(B)). We have revised Section 2.4 to make this clear. We argue that the objective does not require the following:

  1. that Q(A) and Q(B) are represented separately. Q(A) + Q(B) is read at R² = 0.999 (Section 3.2).

  2. that the posterior and value are linearly readable, represented in separate subspaces, and causally used for choice (Sections 3.2 to 3.4).

  3. that the sufficient statistic is the pair of factor posteriors rather than the joint posterior. The joint posterior is readable beyond the factor posteriors (R² 0.894), but steering it with the factor posteriors held fixed moves choice at a slope of 0.000 (Sections 3.2 and 3.4).

  4. that a value forms from the task and encoding alone: under next-token prediction on the identical token stream, no value forms at any layer (R² of 0.003; Section 3.6).

Regarding the positional state-by-state encoding, please see the response to Q1 below.

W2 (Generality) We agree with the reviewer that the present work involves one task and one encoding. The revised work now varies the offer encoding and the reward (Q1 and Q2 below), the structure of the hidden process (a single 9-state HMM, Appendix L), the training objective (Section 3.6), and the model depth (Appendix J). Regarding Qwen3-1.7B adding little, we agree that the test does not reflect natural, plain-language interactions. Instead, we show that the findings are not specific to a small transformer trained from scratch. We reserve naturalistic, plain-language experiments for future work.

W3 (POMDP framing) We agree with the reviewer that stating the study in standard POMDP terms would help to improve clarity. To this end, we have revised Section 2.1. Section 2.2 now provides reasoning for drawing the reward from the posterior (see Q2 below).

W4 (Abstract opening question) We agree with the statement and have altered the abstract.

Q1 (Positional encodings) Following the reviewer's request we have trained a model using a condition-based offer encoding, which removes the positional encoding of the states from the offers: each offer is given as two tokens, one for each of the two hidden Markov chains (Appendix J). We find that the results remain consistent using the conditional encoding: choice follows the target policy (r = 0.999), the factor posteriors are read at layer 0 and the offer values downstream of them (R² of 0.994), and the value steer moves choice at a slope of 0.949. One difference was observed: the posterior steer moves choice at slopes of 0.623 and 0.602 for each factor, against 0.979 and 0.982 in the toy model. This is still well above the off-plane controls (0.073 and 0.093). We therefore conclude that the posterior-value organization we observe does not depend on the state-by-state positional encoding of the offers. Under this design, we cannot conclude whether the factor posteriors come from the structure of the process that generates the clues, or from the offers naming the factors.

Q2 (Reward drawn from posterior) We agree with the reviewer that the motivation for using a reward drawn from the posterior was not clearly stated. Under the standard reward, the expected reward is the payoff mass, m(X) = P(X) Q(X). That is, the probability that the chosen offer holds, multiplied by how good the offer would be if it did hold. We instead want to know how much the model values the offer if it holds, separately from how likely it is to hold. We have clarified this motivation in Section 2.2.

Following the reviewer's suggestion, we trained an additional model using the standard reward. We found that the structure remained intact: the model reaches the target policy (r = 0.999), the posterior forms and can be causally manipulated at layer 0, and value forms at layer 3 and carries choice (steering slope of 0.705, lower than the toy model's 0.944). The two payoff masses are read less precisely than the toy model's offer values (R² of 0.895 against 0.999). Thus, the posterior-value structure holds under the standard reward, though the value of each offer is represented less precisely than in the toy model (Appendix J, Table 5).

Decision-theoretic interpretation. We agree with the reviewer that the decision-theoretic interpretation should not claim more than the evidence supports. Section 4 treats the three properties we test as necessary but not sufficient for attributing belief and desire. We have revised it to state that we do not claim that the network has beliefs or desires.

Thank you for the detailed rebuttal and the new experiments. They resolve my concrete concerns: the encoding, the reward, the framing and the overclaiming. They also add one informative result, the factorized posterior, and I have raised my informativeness score accordingly. The central limitation, however, stands: in a tiny, fixed, one-step world, the organization found is the natural solution, and the study remains a careful case study rather than a more general finding. I therefore keep my rating, but I would not oppose acceptance.

We thank the reviewer for taking the revised manuscript into consideration. We are glad to hear that the reviewer felt their concrete concerns were addressed and found the work to be more informative. We also appreciate that the reviewer does not oppose acceptance. We are glad to provide any further clarifications.

Reviewer_e1tWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the properties of Transformers by investigating a question like “A network could form a posterior over hidden states and value its options under that posterior, or it could map observations to choices with no such intermediate structure.”

Critical strengths
  • The paper is well structured and detailed experimental results are conducted under specific settings.
Critical weaknesses
  • The authors train a transformer to choose between two sets of latent states, where each is a three-state Markov chains. The motivation for this kind of selection is not clear. Moreover, it’s not clear if this specific setting could be extended to general cases.
  • Again, the authors train a four-layer transformer (see page 4), where the motivation and generalization of this setting are not clear.
  • The experimental results lack ablation study and comparison to existing methods/works.
  • It is not clear how the conclusion regarding transformer could be used in practice.
Questions for authors

See above the weaknesses.

3 comments

We thank the reviewer for considering our work and for their thoughtful comments, and we are glad they found the paper well structured and the experiments detailed. We have uploaded a revised PDF and have addressed each of the reviewer's comments point by point below.

W1 (hidden Markov chain motivation) We agree with the reviewer that the motivation for the hidden Markov chains used was not clearly stated. The Mess3 processes used in this work have posterior geometry which is known in closed form and its fractal structure has been established to form in the residual stream of transformers under next-token prediction (Shai et al. 2024, 2026). We used two hidden Markov chains in order to have posterior geometry which can be checked against the previous literature while maintaining a rich enough state space to form offer masks, with 9 states, allowing us to build 511 unique offers. However, the findings of this work extend to circumstances in which the data is generated from a single hidden Markov process. We have revised the manuscript to include a model trained on a one-chain hidden Markov model and we find the same organization: the posterior is read at R² = 0.940 and the offer values at 0.992, steering the posterior moves choice at a slope of 0.872 against an off-plane control of 0.147, steering the value at 0.961 against −0.008, and inserting or removing the value-plane component of a posterior steer gives 0.950 and 0.063 (Appendix L). In regard to extending these findings to general cases, please see the answer under W4.

W2 (model architecture motivation) The reviewer was correct to point out that the motivation for the transformer architecture we used was not properly stated. The transformer depth, residual width, and MLP width were chosen to match Shai et al. 2024. We have revised the manuscript to test multiple transformer depths, testing models with 2, 3, and 8 layers (Appendix J). We find the same organization forms for every model tested: the posterior forms at layer 0 and the offer values downstream of it, with posterior steers between 0.925 and 0.982 against off-plane controls at or below 0.113, and value steers between 0.887 and 0.972 against controls at or below 0.010. Furthermore the results were replicated in Qwen3, which has 28 layers and a residual width of 2048.

W3 (ablation and comparison to existing work/methods) The revised manuscript includes multiple ablations: the objective is varied (Section 3.6), the factored vs joint representation (Section 3.4), the depth (Appendix J), the initialization seeds (Appendix J), the hidden Markov process (Appendix L), and the model architecture (Qwen3, Section 3.5). For each of these, all other settings are held the same. We find the posterior-value organization forms in every case where the network is rewarded according to the value of the offer it chooses. This work used standard mechanistic interpretability methods (steering, probing) to understand the network's representations. The closest paper is Shai et al. 2026; the revised manuscript now includes a model trained only by next-token prediction as a more direct comparison with that work (Section 3.6). Under next-token prediction on the identical stream, the posterior representation forms while no value representation forms at any layer.

W4 (practical use of findings) With this work, we aim to take a first step towards a precise mechanistic account of how transformers make decisions. Using an artificial task, we were able to compute ground truth decision-relevant quantities in order to test them against theory. With this work, we aim to extend the prior state of the art on how a neural network represents the world to understand how it might act using these representations. We aim to extend these findings to more naturalistic settings in future work.

Thanks for the great effort in revising the manuscript. Even though the score is not increased now from me, I would not object it if the decision tends to be acceptance.

We thank the reviewer for taking the time to review the revised manuscript. We appreciate that the reviewer is open to acceptance, and we are glad to provide any further clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.