- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether transformers trained on a synthetic decision task form separable representations of posterior beliefs and action values. In the task, exact posteriors and offer values are analytically computable. The authors train a small transformer with reward-only REINFORCE and show that both quantities are linearly decodable, occupy largely distinct subspaces, generalize to held-out offers, and causally affect choices under activation steering. They replicate the main findings in Qwen3-1.7B-Base trained on the same symbolic task. The paper relates these results to decision-theoretic notions of belief and desire.
Critical strengths
-
Clean ground-truth setup. The synthetic task makes both posterior and offer value exactly computable, enabling unusually controlled probing and intervention analyses.
-
Causal evidence beyond probing. The paper includes steering and mediation-style insertion/removal experiments, which make the evidence stronger than linear decodability alone.
-
Interesting link between posterior geometry and choice. Extending prior posterior-geometry results from prediction to a decision setting is a natural and useful direction.
Critical weaknesses
-
The claims are stronger than the setting supports. The results are framed as evidence for transformer posterior/value representations relevant to decision making, but all main evidence comes from a highly engineered toy task where posterior-to-value computation is the natural solution.
-
Limited novelty over prior work. The posterior geometry largely follows prior HMM work, and the probing/steering tools are standard. The main contribution is their combination in a new synthetic task, which feels incremental.
-
Weak external validity. The Qwen3 experiment uses the same symbolic toy task, not natural-language decisions or realistic preference/reward settings. It shows scale replication, but not generality to real LLM behavior.
-
The reward-only framing is overstated. Payoffs and offer masks are explicitly given in context, and each decision is a contextual bandit. The task avoids long-horizon credit assignment and does not show that utilities are learned from rich interaction.
-
Separable-subspace evidence depends on probe construction. The claimed separation relies on ground-truth-fitted linear probes and covariance-based subspaces. It is unclear how robust this is to alternative probe definitions, task variants, or more distributed/nonlinear representations.
-
Held-out generalization is too small. Generalization is tested in a nine-state world with only 511 possible nonempty masks and 64 held-out masks. This is useful but too limited to support broad claims.
Questions for authors
-
Stronger evidence is needed that the posterior/value organization appears beyond this synthetic task, ideally in more naturalistic language-based decision settings.
-
More discussion is needed on whether the task effectively bakes in the desired posterior-to-value decomposition.
-
Additional baselines would help show whether this organization is transformer-specific, objective-induced, or simply task-induced.
-
Please test robustness to probe construction, subspace definitions, read/write layer choices, and nonlinear or distributed alternatives.
-
The Qwen3 experiment should be better justified as evidence about LLMs, since it is still trained on the same symbolic toy task.
-
Larger and harder generalization tests would strengthen the paper, e.g., larger state spaces, more factors, unseen payoff structures, or held-out structural classes.
We thank the reviewer for their thoughtful feedback and for recognizing the value of our ground-truth setup, causal evidence, and relevance to prior literature. Below, we address their points and the corresponding revisions to the manuscript.
W1 (Claims stronger than evidence supports) We chose a toy decision task for the reason the reviewer lists as a strength: the posterior and offer value can be computed in closed form. Our claims are in regard to this setting, rather than a statement about how transformers behave in general. We have adjusted the claims in the revised manuscript to clarify this, and the limitations paragraph of the Discussion has been extended.
W2 (Limited novelty) We agree with the reviewer that the probing and steering methodology is standard and that the posterior geometry was established by Shai et al. (2024, 2026) under next-token prediction. In this work, we extend this to a setting in which the model is rewarded based on choices, with no next-token prediction, and find that the same posterior geometry forms. Furthermore, to our knowledge, no prior study has characterized posterior and value representations in a single network against ground truth. We aim to extend prior work showing how posterior geometry is represented towards understanding how these representations may be used in practice.
W3, Q1, Q5 (External validity of the Qwen3 experiment) We agree with the reviewer that the Qwen3 experiment does not reflect natural, plain-language interactions. Instead, we show that the findings are not specific to a small transformer trained from scratch: the results replicated in Qwen3, which has 28 layers and a residual width of 2048. We aim to extend these findings to more naturalistic settings in future work.
W4 (Reward-only framing overstated) The reviewer is correct to point out that the task is a contextual bandit, and that the offers and state payoffs are given in context rather than learned from reward. By "reward alone", we meant that the only training signal the model gets is from the sampled reward, and neither the posterior nor the value is directly supervised. To make this clear, we have revised the manuscript to say "choice reward". We leave settings with long-horizon credit assignment to future work.
W5, Q4 (probe construction, subspace definitions, and layer choices) We agree with the reviewer and have added new analyses to the revised manuscript. Using one and two layer MLP probes, we find no evidence that additional information is represented nonlinearly (Appendix D). We now show that the joint posterior is decoded beyond what can be explained by the factor posteriors (R² of 0.894 against 0.761; Section 3.2). However, using a causal ablation intervention, we found that the joint representation is not used for making decisions (slope of 0.000; Section 3.4). Steering through principal components of the activations, rather than subspaces fit to the ground truth, gives the same result (value slope of 0.944; posterior slopes of 0.997 and 0.991; Section 3.4). Appendix E shows the probing and steering results across layers. The linear probe accuracy changes little under modifications to the probes' regularization, training-set size and readout position (Appendix D).
W6, Q6 (held-out generalizations) We agree with the reviewer that the state space is small and have added two generalization tests. First, we trained a model in which a structural class of offers was held out, the 27 product sets. The organization was unchanged, and on the 27 never-offered product sets the offer value is read at a median R² of 0.999 (Appendix J). We also tested a set of payoff cards that were never seen during training, on which the offer values are read at R² = 0.999 (Appendix G). We reserve testing larger state spaces and more factors for future work.
Q2 (Baked in structure) Following the reviewer's suggestion, we expanded Section 2.4 to discuss what is entailed by the task. The task requires a sufficient statistic of the clues, and a choice logit equal to β(Q(A) − Q(B)), but it does not require the organization we report. Q(A) and Q(B) need not be represented separately, yet Q(A) + Q(B) is read at R² = 0.999 (Section 3.2). The task also does not force separate subspaces, linear readability, or causal use of the representations, but we find these nonetheless (Sections 3.2 to 3.4).
Q3 (Additional baselines) We now train a network using only next-token prediction, without any choice reward, and find that the posterior representation forms, but the value representations do not (Section 3.6). We also find the same organization in a model trained using a non-factored nine-state Markov chain (Appendix L), in a setting where offers are given as conditions per chain (Appendix J), and in models with 2, 3 and 8 layers (Appendix J). We did not test architectures other than transformers.