- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses two interesting questions for POMDP using transformers. First, can reward-only RL learn Bayesian beliefs. Through affine probes, Bayesian beliefs can be decoded from transformer representations after training. Second, if the transformer does not learn the full belief, does it selectively discard information that is irrelevant to optimal control? The experiment shows that under linear decoding, the reward-only training selectively preserves belief information that is related to optimal control.
Critical strengths
The design of the belief-quotient ladder clearly aligns with the research questions stated in the introduction. This design shows how control requirements affect learned belief representations. In particular, Section 4.3 provides a meaningful comparison of how well the transformer's hidden representations encode the full-filter marginal and the belief of an exact two-state quotient POMDP. This goes beyond previous studies by explicitly investigating task-dependent belief coarsening.
Critical weaknesses
-
Limitations of linear probing. The analysis relies on affine probes to evaluate how well the belief representations are learned. However, linear probing cannot fully characterize the information contained in the hidden representations.
-
Limited novelty compared with previous studies. The findings are closely related to previous RNN-based studies such as Lambrechts et al. (2022) and Hennig et al. (2023). In particular, the experiment in Section 5 does not show results substantially beyond previous findings of Lambrechts et al. (2022). Since the second factor is independent and does not contribute to the reward, it is expected that reward-based learning would mainly focus on the first factor. From the perspective of neural networks as feature extractors, this result does not provide much additional insight.
Questions for authors
-
From a representation learning perspective, both RNN and Transformer learn hidden representations from history. If we view both architectures as feature extractors, can the author further explain the difference between this paper and previous studies.
-
Would an RNN trained with the policy-gradient objective show similar belief coarsening? Such a comparison could help clarify whether the findings are specific to Transformers or reflect a more general property of reward-driven representation learning.
4 comments
Response to Reviewer MDW3 (2/2): Q2
Q2 — Comparison with recurrent policies.
We performed an additional rebuttal comparison with two GRU policies trained using the same policy-gradient recipe and environment-step budget, with approximately matched parameter counts. One uses two recurrent layers with 128 hidden units; the other uses four layers with 92 units. Each configuration has approximately 200,000 parameters.
Both GRUs show the same qualitative preference for the coarse-filter target in Reward-State Quotient. The following are held-out probe errors, , for the first training seed of each architecture:
| Architecture | Full-belief error | Coarse-filter error |
|---|---|---|
| Transformer | 0.126 | 0.016 |
| GRU with 128 hidden units | 0.048 | 0.003 |
| GRU with 92 hidden units | 0.138 | 0.005 |
In Full-State Quotient, both GRUs decode the full belief accurately, with errors of 0.017–0.018. Their ability to expose the finer belief therefore depends on the task, as in the transformer comparison.
These results suggest that the observed preference for coarse belief targets is not exclusive to attention-based encoders. They are preliminary: each recurrent configuration has one training seed, and the probes measure affine accessibility rather than all stored information or causal use. The targets also differ in dimensionality, so their error ratio is a descriptive accessibility comparison, not a proof of a unique internal quotient filter.
We therefore view the RNN comparison as evidence that the phenomenon can occur in both recurrent and attention-based policies, while leaving its broader dependence on architecture, capacity, and optimization open.
Thank you for the response. It remains unclear how GRUs and Transformers differ as feature extractors in these tasks. A theoretical analysis or more targeted experiments are needed to clarify how architecture affects the learned belief representations. I maintain my Weak Reject rating.
Thank you for the follow-up.
We agree that how architecture shapes learned belief representations is an interesting question, and we would not claim to have answered it. We would respectfully suggest, though, that it is a different question from the one this paper asks. Our claims concern which beliefs a reward-trained transformer agent represents and acts on, and how this depends on the task. None of them is a claim about transformers as opposed to recurrent networks, and none depends on such a comparison. As we said in our first reply, a change of architecture is not part of our contribution.
We studied transformers because the belief-geometry methods we build on (Shai et al., 2024) were developed for residual-stream representations, and because transformer policies are increasingly used as the architecture of agents. A systematic account of architectural effects, whether theoretical or experimental, would need to vary capacity, depth, context length and optimizer across architectures. That is a study in its own right, and we believe it is better treated as future work rather than as a condition on the claims made here.
Your review raised two weaknesses: what affine probes can establish (W1), and novelty relative to Lambrechts et al. and Hennig et al. (W2). We would be grateful to know whether the interventions and the revised positioning address these, or whether there is something further we can clarify.
Response to Reviewer MDW3 (1/2): W1, W2, Q1
Thank you for highlighting both the task-dependent coarsening analysis and the need to distinguish it more clearly from previous studies. We agree with your concerns about the interpretation of affine probes. This feedback helped us clarify the contribution and scope of the paper.
W1 — What affine probes establish.
We agree that an affine probe cannot fully characterize a representation: a belief encoded nonlinearly would be missed, and Section 6.2 states this as a limitation. Nonlinear probing is an interesting direction for future work. Our conclusions do not rest on a complete characterization, however. Moreover, linear probing proves to be a useful tool for understanding what drives our policies' behaviour and even controlling it ourselves; these are among the methodological uses of linear probes that we contribute.
For instance, in RockSample, the probe fails to recover one rock's belief in each of the five lower-performing agents (R2<0), and behaviour agrees: those agents essentially never collect that rock when it is Good (Section 5.1). In the Belief-Quotient Ladder and Reward Relevance experiments, the affine probe decoders themselves drive the interventions: exchanging the decoded probabilities of states 1 and 2 changes 46% of Full-State Quotient actions, and steering the decoded belief raises reward on a changed task from 0.901 to 1.042 without weight updates (Sections 3.4 and 4.2). In RockSample, a linear edit along directions estimated from counterfactual sensor histories reduces collection of a Good rock from 98.0% to 1.4% while other Good rocks remain at 97.6% (Section 5.2).
These results rest on methodological uses of linear probes that go beyond a single goodness-of-fit score. Action-conditioned Bayesian targets extend belief probing to closed-loop control, where the agent's own actions change the dynamics (Section 3.1; Appendix A). The probes are used comparatively: we track how probe error changes from each network's own initialization on identical histories, within greedy-action classes, and across candidate belief targets such as the full belief and a coarse filter (Section 3.3; Appendix G). The fitted decoders then serve as the diagnostics and the basis for the activation interventions described above, so that probe readings are checked against behaviour.
W2 and Q1 — Contribution beyond earlier RNN studies.
We agree that the broad findings of belief-like representation and selective retention of relevant information are established by the earlier RNN studies. The revised introduction acknowledges this directly, and the earlier priority claim concerning PPO has been removed. Our contribution does not depend on treating an architecture change as a new general phenomenon.
The more specific contribution is to examine which candidate belief geometry is affinely accessible, how training changes that geometry beyond current-action identity, and which belief-related interventions affect behaviour.
We also agree that the independent-factor experiment, considered alone, is an extension of Lambrechts et al. rather than an entirely new phenomenon. Its additional control is that the second factor remains observable, dynamic, and controllable while its reward contribution is removed. The observation model, transition dynamics, and available actions are held fixed. The irrelevant factors in Lambrechts et al.'s experiment are independent Gaussian random walks whose dynamics do not depend on the agent's actions. Our manipulation therefore separates reward relevance from uncontrollability.
The revision adds behavioural applications of the analysis. In RockSample, poor affine access to one rock's belief identifies a specific neglected object in the lower-performing agents. Separately, interventions in successful agents selectively change whether a particular rock is collected. In the changed-reward experiment, an external controller uses belief-related activations to redirect frozen policies. These results make the representation analysis useful for diagnosis and controlled behavioural modification, while retaining the scope qualifications described in the paper.
The candidate-target comparison also requires care. A coarse filter and a full-observation posterior condition on different information, and better affine fit does not uniquely identify the implemented computation. We therefore interpret these comparisons together with the matched-initialization tests and interventions, rather than treating probe accuracy alone as a complete mechanistic explanation.
Q1. See our combined response to W2 and Q1 for the distinction from previous RNN studies.