acceptodds
Under review as a conference paper at ICLR 2027

Deep reinforcement learning agents represent reward-dependent belief geometry

Abstract

Bayesian beliefs are sufficient statistics of the history of a partially observed process, and classical separation results show that optimal control can be attained using these beliefs. However, it is unclear whether belief-like representations actually emerge in model-free deep reinforcement learning (RL), which trains policies not for prediction but to maximize reward. To address this question, we study transformer policies trained by policy gradients and fit affine probes into their residual streams, scanning for Bayesian posteriors. Experiments show that reward alone suffices to recover beliefs, even in the absence of latent-state supervision or auxiliary losses. However, we also find that the full belief geometry is not always what the agent learns. Using a family of tasks, in which the reward–action structure requires progressively finer distinctions between latent states, we show that distinctions that are irrelevant for control and reward are selectively lost, and that the learned representation is better described by the beliefs of a coarse-grained POMDP than by the full environmental posterior. Changing the reward leads to the same effect: only reward-relevant latent factors are accounted for within the belief geometry. Taken together, these results suggest a learning-dependent version of the separation principle, in which the environment determines how beliefs are updated while the reward determines which beliefs are worth maintaining.

Primary area:
interpretability and explainable AI
Keywords:
reinfocement learning, belief geometry, mechanistic interpretability, optimal control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MDW3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses two interesting questions for POMDP using transformers. First, can reward-only RL learn Bayesian beliefs. Through affine probes, Bayesian beliefs can be decoded from transformer representations after training. Second, if the transformer does not learn the full belief, does it selectively discard information that is irrelevant to optimal control? The experiment shows that under linear decoding, the reward-only training selectively preserves belief information that is related to optimal control.

Critical strengths

The design of the belief-quotient ladder clearly aligns with the research questions stated in the introduction. This design shows how control requirements affect learned belief representations. In particular, Section 4.3 provides a meaningful comparison of how well the transformer's hidden representations encode the full-filter marginal and the belief of an exact two-state quotient POMDP. This goes beyond previous studies by explicitly investigating task-dependent belief coarsening.

Critical weaknesses
  1. Limitations of linear probing. The analysis relies on affine probes to evaluate how well the belief representations are learned. However, linear probing cannot fully characterize the information contained in the hidden representations.

  2. Limited novelty compared with previous studies. The findings are closely related to previous RNN-based studies such as Lambrechts et al. (2022) and Hennig et al. (2023). In particular, the experiment in Section 5 does not show results substantially beyond previous findings of Lambrechts et al. (2022). Since the second factor is independent and does not contribute to the reward, it is expected that reward-based learning would mainly focus on the first factor. From the perspective of neural networks as feature extractors, this result does not provide much additional insight.

Questions for authors
  1. From a representation learning perspective, both RNN and Transformer learn hidden representations from history. If we view both architectures as feature extractors, can the author further explain the difference between this paper and previous studies.

  2. Would an RNN trained with the policy-gradient objective show similar belief coarsening? Such a comparison could help clarify whether the findings are specific to Transformers or reflect a more general property of reward-driven representation learning.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MDW3 (1/2): W1, W2, Q1

Thank you for highlighting both the task-dependent coarsening analysis and the need to distinguish it more clearly from previous studies. We agree with your concerns about the interpretation of affine probes. This feedback helped us clarify the contribution and scope of the paper.

W1 — What affine probes establish.

We agree that an affine probe cannot fully characterize a representation: a belief encoded nonlinearly would be missed, and Section 6.2 states this as a limitation. Nonlinear probing is an interesting direction for future work. Our conclusions do not rest on a complete characterization, however. Moreover, linear probing proves to be a useful tool for understanding what drives our policies' behaviour and even controlling it ourselves; these are among the methodological uses of linear probes that we contribute.

For instance, in RockSample, the probe fails to recover one rock's belief in each of the five lower-performing agents (R2<0R^2 < 0), and behaviour agrees: those agents essentially never collect that rock when it is Good (Section 5.1). In the Belief-Quotient Ladder and Reward Relevance experiments, the affine probe decoders themselves drive the interventions: exchanging the decoded probabilities of states 1 and 2 changes 46% of Full-State Quotient actions, and steering the decoded belief raises reward on a changed task from 0.901 to 1.042 without weight updates (Sections 3.4 and 4.2). In RockSample, a linear edit along directions estimated from counterfactual sensor histories reduces collection of a Good rock from 98.0% to 1.4% while other Good rocks remain at 97.6% (Section 5.2).

These results rest on methodological uses of linear probes that go beyond a single goodness-of-fit score. Action-conditioned Bayesian targets extend belief probing to closed-loop control, where the agent's own actions change the dynamics (Section 3.1; Appendix A). The probes are used comparatively: we track how probe error changes from each network's own initialization on identical histories, within greedy-action classes, and across candidate belief targets such as the full belief and a coarse filter (Section 3.3; Appendix G). The fitted decoders then serve as the diagnostics and the basis for the activation interventions described above, so that probe readings are checked against behaviour.

W2 and Q1 — Contribution beyond earlier RNN studies.

We agree that the broad findings of belief-like representation and selective retention of relevant information are established by the earlier RNN studies. The revised introduction acknowledges this directly, and the earlier priority claim concerning PPO has been removed. Our contribution does not depend on treating an architecture change as a new general phenomenon.

The more specific contribution is to examine which candidate belief geometry is affinely accessible, how training changes that geometry beyond current-action identity, and which belief-related interventions affect behaviour.

We also agree that the independent-factor experiment, considered alone, is an extension of Lambrechts et al. rather than an entirely new phenomenon. Its additional control is that the second factor remains observable, dynamic, and controllable while its reward contribution is removed. The observation model, transition dynamics, and available actions are held fixed. The irrelevant factors in Lambrechts et al.'s experiment are independent Gaussian random walks whose dynamics do not depend on the agent's actions. Our manipulation therefore separates reward relevance from uncontrollability.

The revision adds behavioural applications of the analysis. In RockSample, poor affine access to one rock's belief identifies a specific neglected object in the lower-performing agents. Separately, interventions in successful agents selectively change whether a particular rock is collected. In the changed-reward experiment, an external controller uses belief-related activations to redirect frozen policies. These results make the representation analysis useful for diagnosis and controlled behavioural modification, while retaining the scope qualifications described in the paper.

The candidate-target comparison also requires care. A coarse filter and a full-observation posterior condition on different information, and better affine fit does not uniquely identify the implemented computation. We therefore interpret these comparisons together with the matched-initialization tests and interventions, rather than treating probe accuracy alone as a complete mechanistic explanation.

Q1. See our combined response to W2 and Q1 for the distinction from previous RNN studies.

AuthorsOpenReview ↗

Response to Reviewer MDW3 (2/2): Q2

Q2 — Comparison with recurrent policies.

We performed an additional rebuttal comparison with two GRU policies trained using the same policy-gradient recipe and environment-step budget, with approximately matched parameter counts. One uses two recurrent layers with 128 hidden units; the other uses four layers with 92 units. Each configuration has approximately 200,000 parameters.

Both GRUs show the same qualitative preference for the coarse-filter target in Reward-State Quotient. The following are held-out probe errors, 1−R21-R^2, for the first training seed of each architecture:

ArchitectureFull-belief errorCoarse-filter error
Transformer0.1260.016
GRU with 128 hidden units0.0480.003
GRU with 92 hidden units0.1380.005

In Full-State Quotient, both GRUs decode the full belief accurately, with errors of 0.017–0.018. Their ability to expose the finer belief therefore depends on the task, as in the transformer comparison.

These results suggest that the observed preference for coarse belief targets is not exclusive to attention-based encoders. They are preliminary: each recurrent configuration has one training seed, and the probes measure affine accessibility rather than all stored information or causal use. The targets also differ in dimensionality, so their error ratio is a descriptive accessibility comparison, not a proof of a unique internal quotient filter.

We therefore view the RNN comparison as evidence that the phenomenon can occur in both recurrent and attention-based policies, while leaving its broader dependence on architecture, capacity, and optimization open.

Thank you for the response. It remains unclear how GRUs and Transformers differ as feature extractors in these tasks. A theoretical analysis or more targeted experiments are needed to clarify how architecture affects the learned belief representations. I maintain my Weak Reject rating.

Thank you for the follow-up.

We agree that how architecture shapes learned belief representations is an interesting question, and we would not claim to have answered it. We would respectfully suggest, though, that it is a different question from the one this paper asks. Our claims concern which beliefs a reward-trained transformer agent represents and acts on, and how this depends on the task. None of them is a claim about transformers as opposed to recurrent networks, and none depends on such a comparison. As we said in our first reply, a change of architecture is not part of our contribution.

We studied transformers because the belief-geometry methods we build on (Shai et al., 2024) were developed for residual-stream representations, and because transformer policies are increasingly used as the architecture of agents. A systematic account of architectural effects, whether theoretical or experimental, would need to vary capacity, depth, context length and optimizer across architectures. That is a study in its own right, and we believe it is better treated as future work rather than as a condition on the claims made here.

Your review raised two weaknesses: what affine probes can establish (W1), and novelty relative to Lambrechts et al. and Hennig et al. (W2). We would be grateful to know whether the interventions and the revised positioning address these, or whether there is something further we can clarify.

Reviewer_YXoqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate whether transformer policies that are trained with model-free PPO spontaneously develop internal representations of Bayesian belief state geometry in Partially Observable Markov Decision Processes (POMDPs) without explicit belief supervision or auxiliary prediction losses. Their results confirm that this is the case in

Critical strengths

Previous research only tested whether transformers track hidden probabilities when passively watching a stream of tokens whereas this paper extends that test to interactive reinforcement learning agents (so their own actions change what happens next).

The authors empirically demonstrate that exact Bayesian belief simplex geometry (R2>0.996R^2 > 0.996) can emerge linearly in the residual streams of transformers trained purely with model-free policy gradients (PPO).

The authors have designed a suite of environments to empirically demonstrate that the agent learns to ignore the irrelevant details and only tracks a simplified version of the belief state that is strictly needed to get the reward. This will be reusable in future experiments.

Critical weaknesses

None

Questions for authors

The authors only probe the last layer right before the action readout. Earlier layers might still track the full environment state and only drop irrelevant details right before picking an action. Probing other layers would clarify where the simplification actually happens.

Can the author expand a bit more on why the 1-R^2 is so high at 0 environment steps?

1 comment
AuthorsOpenReview ↗

Response to Reviewer YXoq

Thank you for the positive assessment and for these two questions. They helped us distinguish what the probes establish about a particular readout from what can be concluded about the network as a whole.

Q1 — Representations in earlier layers.

We agree, and have probed all five sites of the Belief-Quotient Ladder agents: after each of the four transformer blocks and after the final layer normalization, both at initialization and after training, for all 15 training seeds of each condition. The simplification does not happen at the readout. In the Reward-State Quotient agent the quotient belief is already formed after the first block (probe error 1−R21-R^2 of 0.04 there and 0.02 at every later site, against 0.29–0.37 at initialization), while at no depth does the agent resolve the two nonreward states better than its untrained network (0.22–0.43 after training, against 0.13–0.21 at initialization). The Full-State Quotient agent instead resolves the full belief from the first block onward (error falls from 0.20 to 0.08 after the first block, and is 0.03–0.04 at later sites). Steering in RockSample points the same way: the belief that drives behavior is already present after the first block, where exchanging the evidence about one rock reverses whether the agent samples it (a Good rock is then sampled in 1.4% of episodes and a Bad rock in 91.8%) while leaving the other rocks unaffected, and the same edit is less effective from the third block onward (10.9% and 51.8%).

Q2 — Probe performance at initialization.

At zero environment steps, the policy is untrained, but the affine probe is still fitted to data. The untrained transformer supplies a random feature map of recent observations and actions. Those inputs already constrain the Bayesian posterior, so a fitted readout can recover some belief information before the policy has learned from reward. The quantity 1−R21-R^2 measures the remaining normalized error: zero is perfect decoding, and one matches prediction by the test-set mean. Its initial value therefore depends on the environment, architecture, and available context. A high initial R2R^2 does not imply that the agent has learned Bayesian inference.

Instead, we use the change relative to a matched initialization. In Appendix G, trained and initialized networks receive identical histories and use the same probe targets and fitting and test rows. For instance, in Reward-State Quotient, the reward-state coordinate rises from 0.902 to 0.968, while the distinction between states 1 and 2 falls from 0.875 to 0.798. These comparisons quantify what training changes beyond the information already available in the initialized features. The interventions agree with this pattern in the same agent. For instance, editing the reward-state coordinate, which training refines, redirects behaviour, whereas exchanging the beliefs in states 1 and 2, whose accessibility falls, changes fewer than 1% of actions (Section 3.4).

Reviewer_DmAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether Bayesian belief-like representations emerge in transformer policies trained with model-free policy gradients, with no prediction loss or latent-state supervision, and which parts of the belief are retained. It extends Shai et al.'s (2024) affine-probing methodology to closed-loop control by constructing action-conditioned Bayesian targets, and reports three experiments on variants of the Mess3 process.

(i) In a contextual-bandit token-prediction task, a transformer trained with PPO on 0/1 reward yields a residual stream from which the full belief simplex is affinely decodable (R² ≈ 0.996), matching a cross-entropy control. (ii) In a "Belief-Quotient Ladder" of three POMDP conditions, where the optimal action depends on progressively finer latent-state distinctions, full-belief decodability tracks the control requirement; in the Reward-State Quotient condition, an exact two-state quotient filter fits the representation an order of magnitude better than the marginal of the full three-state filter. (iii) In a two-factor environment with fixed dynamics and actions, removing one factor's reward removes the decodability of that factor's belief.

The authors interpret this as a learning-dependent separation principle: environment dynamics determine how beliefs update, while reward determines which beliefs are worth maintaining.

Critical strengths
  • The action-conditioned Bayesian probe targets (Appendix A) are a sensible, reusable extension that makes belief probing applicable to policies, not only to next-token predictors.
  • Section 4.3 is the most interesting part of the paper. Comparing an exact quotient filter against the full-filter marginal on paired histories is a nice diagnostic, and the resulting methodological point (score probes against the task-induced quotient belief, not automatically against the maximally resolved posterior) is the paper's most durable takeaway.
  • Section 5 is a clean design: reward is changed while dynamics, observations and actions are held fixed, with within-seed contrasts across 15 seeds.
  • The hypotheses are stated explicitly up front; the controls (history shuffle, random input, initialization) are appropriate.
Critical weaknesses
  1. The central finding is largely entailed by behavioral success, and in that form it is already published. If a policy attains high return and the return depends on some component of the belief, that component is present in the representation in some form. This follows from success, not from an experiment, and the positive half of the results (control-relevant information is decodable; the ordering in Figure 2a) is of this kind. The negative half (reward-irrelevant factors are not represented beyond what a random network already provides; Section 5) is a genuine observation, but it is the one Lambrechts et al. (2022) already report for recurrent Q-networks in information-theoretic form, and Hennig et al. (2023) already report belief-like representations emerging from RL. Both are cited; the stated delta ("an explicitly geometry-resolved, action-conditioned test") is methodological and narrow. H2 is, in POMDP terms, the content of reward-respecting state abstraction and belief bisimulation (Li et al., 2006; Castro et al., 2009), also cited. The paper does not say what a reader learns beyond this, nor how the finding would help build better agents.

    • Suggestion: state precisely what is claimed beyond "control-relevant information is present and irrelevant information is not", and position the contribution against Lambrechts et al. and Hennig et al. in the introduction, not only in Related Work. The claim in Section 3.2 of a "first example of belief representation arising from a policy trained via PPO" should be scoped: if it is narrowly about PPO it is not informative, and otherwise it overclaims relative to the works above.
  2. The reported probe R² cannot support the claim, because it does not separate what behavior already guarantees from what would be a finding. A successful policy must carry which decision region b_t lies in (the optimal policy is piecewise constant in b), so that part of the belief is decodable by necessity, not by discovery (W1). The non-trivial claim, and the only reading of the Section 2.4 hypothesis with content beyond definitions, is that the network also represents where b_t lies within a region: continuous coordinates that the current action does not need. The global R² lumps the two together, and with emission probability 0.85 beliefs concentrate near the vertices, so the between-region component may dominate. It also does not separate what training added from what a random network over a short window already provides: Figure 5 shows R² ≈ 0.97 at initialization, so the "emergence" is the move from ≈ 0.97 to ≈ 0.996.

    • Suggestion: report the probe R² restricted to timesteps within a single greedy-action class, for the trained network and for the initialized one. Within a class, the action is constant, so decodable variance there is, by construction, variance that behavior did not need; if it is high and improves with training, the belief-geometry claim has non-tautological support. This can be computed from the logged rollouts.
  3. Generality is asserted, not shown; the title promises a law where the evidence is three designed conditions. The ladder's conditions are exactly the three extreme cases (constant / exact one-dimensional quotient / full belief), and the quotient filter of Section 4.3 exists in closed form only because the environment was built for exact lumpability (Appendix C.2). No intermediate or approximately lumpable case is examined, which is where the conjecture would gain generality or fail. H1 as stated ("can emerge") is too weak to be useful: emergence in Mess3 does not imply that it generally occurs, and the practical effectiveness of auxiliary belief and reconstruction losses (Section 6) is itself evidence that reward alone frequently does not suffice; the paper does not engage this tension. Appendix C.3 states that PPO and SAC were also run and converged, yet no probe results are reported for them, and the switch from PPO (Section 3) to REINFORCE (Sections 4 and 5) is unexplained. No theoretical argument is offered that would turn the observations into a principle.

    • Suggestion: (i) perturb the symmetry (e.g., slightly different emission probabilities for states 1 and 2) and report which target the representation tracks: the approximate quotient, the full belief, or neither; (ii) report probes for the PPO and SAC runs already trained; (iii) adjust the title and abstract to the evidence ("can represent", or name the setting).
  4. All evidence is correlational, and the stated implications are not tested. The authors acknowledge that affine fits with correlated targets cannot show that the network executes the quotient recursion or uses the decoded quantity; even one activation intervention along the probe direction would strengthen Section 4.3 considerably. The Discussion's implications for world models and generalization (a quotient-belief agent must relearn its representation when reward changes, whereas an agent with an auxiliary prediction loss need only change its controller) are asserted, not argued or tested. The experiment is a direct extension of Section 5: switch the rewarded factor mid-training and measure adaptation sample-efficiency for reward-only versus reward-plus-auxiliary agents, against the pre-switch probe error on the newly relevant factor. Without it, that paragraph should be reframed as a hypothesis.

  5. The main text is not self-contained. The Mess3 process, the task of Section 3.2 (timing, observation, reward), the ladder environments (transition matrix, emissions, tilt actions), and the action-conditioned probe construction, which is the paper's main methodological contribution, are all in the appendix. The results of Sections 3 and 4 cannot be understood without them. Hyperparameters belong in an appendix; the definition of the problem being solved and of the method being introduced do not.

Questions for authors
  1. Within a single greedy-action class, how much of b_t is affinely decodable, and how does this compare with the initialized network and the shuffle controls? (W2)
  2. What return does a memoryless policy acting on the current token achieve in each ladder condition, and how far above it are the trained agents?
  3. How does the representation behave when lumpability is only approximate, e.g. when states 1 and 2 have slightly different emission probabilities? Which target fits best then?
  4. Why PPO in Section 3 but REINFORCE in Sections 4 and 5? What do the probes look like for the PPO and SAC runs mentioned in Appendix C.3?
  5. Does an activation intervention along the probe direction change behavior in the direction the belief interpretation predicts?
  6. Is the probe extension in Appendix A more than selecting the kernel K^{a,o} by the executed action in Eq. (9)? If it is the paper's main methodological contribution, what is the non-obvious part, and why is it not in the main text?
  7. Please clarify the scope of the "first example ... via PPO" claim in Section 3.2 relative to Hennig et al. (2023) and Lambrechts et al. (2022).
8 comments
AuthorsOpenReview ↗

Response to Reviewer DmAd (1/6): W1, Q7

Thank you very much for the detailed review. The distinction between information implied by successful behaviour and a particular internal belief geometry helped us sharpen the paper. To properly address the comments, we have redesigned the ladder experiment, added the suggested within-action-class analysis, and introduced activation interventions and a standard POMDP benchmark.

W1 and Q7 — Contribution and relationship to prior work.

We agree that successful behaviour alone does not establish a strong representational finding, and that the broad emergence of belief-like representations under reward-based learning was established before this work. We now acknowledge this foundation directly in the introduction and have removed the "first example ... via PPO" claim. Changing the architecture or optimizer is not our main novelty claim.

Accordingly, the revised paper focuses on more specific questions. First, does training refine continuous belief coordinates beyond what can be explained by the current action and the initialized network? Second, which belief coordinates influence behaviour under targeted interventions? Third, can those representations provide a useful means of directing a trained policy? The new within-class analysis and interventions answer these questions together. The RockSample and changed-reward experiments then provide concrete behavioural applications.

We also agree that state abstraction and belief bisimulation provide established theoretical conditions under which a given abstraction preserves control. They do not imply that an end-to-end learner acquires that abstraction or exposes it in affine coordinates. Our results concern this learned realization, with limitations described below.

AuthorsOpenReview ↗

Response to Reviewer DmAd (2/6): W2, Q1

W2 and Q1 — Belief decoding within greedy-action classes.

We ran the analysis you suggested. It exposed a limitation of the original ladder: policies using only the current token obtained essentially the same return as the transformer policies, and the original within-class comparison showed very little change from initialization, below 0.001 in R2R^2. The original experiment therefore did not adequately test whether training develops history-dependent belief geometry.

We consequently redesigned the ladder using more persistent latent dynamics and less informative observations, and reran the experiments with the revised training protocol. The results below refer to this revised experiment, not to additional measurements of the original runs. Appendices C.1 and C.3 specify the revised environment and training protocol; the memoryless comparison is reported in our response to Q2.

For each trained policy, we group timesteps by its greedy action, fit a separate affine probe within each class, and normalize the pooled squared error by the within-class target variance. To isolate the effect of training, we replay exactly the same observation-action histories through the initialized network of the same seed. Histories, targets, action classes, and fitting and test rows are therefore matched.

The following are held-out within-class R2R^2 values averaged over 15 training seeds (Appendix G, Table 8):

ConditionTargetMatched initializationTrained
Full-State QuotientFull belief btb_t0.8400.951
Full-State QuotientReward-state probability ρt\rho_t0.8600.967
Full-State QuotientState distinction δt\delta_t0.7960.916
Reward-State QuotientFull belief btb_t0.8850.865
Reward-State QuotientReward-state probability ρt\rho_t0.9020.968
Reward-State QuotientState distinction δt\delta_t0.8750.798

Here ρt=bt(3)\rho_t=b_t(3) and δt=bt(1)−bt(2)\delta_t=b_t(1)-b_t(2). The Full-State Quotient full-belief improvement is 0.111, with a paired 95% seed-bootstrap interval of [0.104,0.119][0.104, 0.119]. The increases for both Full-State Quotient coordinates, the increase for Reward-State Quotient ρt\rho_t, and the decrease for Reward-State Quotient δt\delta_t each occur in all 15 seeds.

The greedy-action class explains 56% of total belief variance in Full-State Quotient and 38% in Reward-State Quotient. Thus, 44% and 62%, respectively, remain within regions of constant action. The probes recover substantial portions of this remaining variation, and training changes its accessibility in different directions for different coordinates. Refitted shuffle, random-input, and within-class label-permutation controls average between −0.03-0.03 and 0.050.05.

We also tested whether the within-class result merely reflects the policy's confidence in its chosen action. Removing the two residual-stream directions that determine action-logit differences leaves almost all decoding performance intact: Full-State Quotient full-belief R2R^2 is 0.950 instead of 0.951, and Reward-State Quotient ρt\rho_t and δt\delta_t scores are 0.968 and 0.797 instead of 0.968 and 0.798. The logits alone explain substantially less of the belief variation (Appendix G.3).

These findings establish training-dependent geometry beyond the current action label and immediate action output.

AuthorsOpenReview ↗

Response to Reviewer DmAd (3/6): W3, Q3

W3 and Q3 — Approximate lumpability and generality.

We ran the proposed emission perturbation as an additional rebuttal experiment. Starting from Reward-State Quotient, we change the probability of token C to 0.85−κ0.85-\kappa in state 1 and 0.85+κ0.85+\kappa in state 2, rescaling their A/B probabilities to retain their original ratios. Transitions, available actions, and reward remain unchanged. For positive κ\kappa the control problem no longer has an exact quotient: the original two-state recursion is only an approximate filter. We keep that recursion fixed and compare its scalar target ct[B]c_t[\mathcal{B}] with the exact reward-state posterior ρt\rho_t. We trained three seeds at each perturbation with two algorithms: the critic-free REINFORCE of the paper, and PPO, which adds a learned critic, here with an entropy bonus annealed to zero. The held-out results are:

κ\kappaREINFORCE: ct[B]c_t[\mathcal{B}] R2R^2REINFORCE: ρt\rho_t R2R^2REINFORCE: rewardPPO with entropy bonus: ct[B]c_t[\mathcal{B}] R2R^2PPO with entropy bonus: ρt\rho_t R2R^2PPO with entropy bonus: rewardReference reward: coarseReference reward: full
00.9800.9790.4240.9980.9980.4570.4540.455
0.0250.9860.9810.4020.9960.9970.4520.4480.451
0.0500.9790.9620.3900.9930.9960.4390.4290.438
0.0750.9760.9290.3570.9890.9920.4180.3950.416
0.1000.9860.8950.3240.9810.9710.3830.3490.384

R2R^2 is for the trained network, and reward is per step under the greedy policy, without the bonus. The references are threshold controllers that act on the coarse filter and on the full posterior. They are not certified optima; the return gap between them grows from 0.3% to 9.1%.

Where the quotient is exact, both algorithms learn it. At κ=0\kappa=0 both decode the quotient target, and both make the distinction between states 1 and 2, which control does not need, less accessible than at initialization (R2R^2 falls from 0.86 to 0.71 under REINFORCE and from 0.86 to 0.76 under PPO).

Where it is only approximate, the two algorithms part ways. PPO agents follow the finer belief as it becomes useful. They match the full-posterior reference at every κ\kappa, to within 0.002, decode ρt\rho_t at 0.97 or better, and refine the distinction between states 1 and 2 in step with its value for control: relative to initialization its R2R^2 changes by −0.09-0.09, +0.02+0.02, +0.09+0.09, +0.12+0.12 and +0.12+0.12 as κ\kappa grows. REINFORCE agents stay with the coarse filter. As the targets separate, ct[B]c_t[\mathcal{B}] remains accurately decoded while ρt\rho_t does not: at κ=0.10\kappa=0.10 training raises the former from 0.965 to 0.986 and leaves the latter at 0.89. These agents also earn 0.02–0.05 less than the coarse reference at every κ\kappa, including κ=0\kappa=0.

What this means for the hypothesis. You asked which target the representation tracks when lumpability is only approximate. The answer depends on whether the agent reaches the reward that the finer belief makes available. The PPO agents give the graded case: the coordinate that exact lumpability makes redundant is suppressed at κ=0\kappa=0 and refined progressively as the perturbation gives it value for control. In these three-seed runs, the dependence of the learned belief on control requirements is therefore not confined to exactly lumpable environments. The REINFORCE agents are a failure case. They keep the approximate quotient where it no longer suffices. Because they fall short of the reference at κ=0\kappa=0 as well, we cannot tell whether the coarse representation limits their return or follows from incomplete optimization. Nor does the comparison show what PPO contributes: it adds a critic and an entropy bonus together, so the experiment does not show which term matters, and it does not separate a changed objective from better optimization of the same reward. We therefore do not claim that reward-only training reliably finds a representation sufficient for optimal control. What we do claim is stated by the revised hypotheses in Section 2.4: training refines chiefly the belief coordinates that control requires (H1), and the policy acts on those coordinates (H2). H2 is new in this revision. The previously submitted version contained no interventions; the steering experiments in the ladder, the two-factor environment and RockSample (Sections 3.4, 4.2 and 5.2) now show that the refined coordinates are used by the policy and can be steered to redirect behaviour. Our response to Q4 reports that both claims also held under PPO and SAC, and that only the further claim, that coordinates the task does not require are left unrefined, is specific to the critic-free setting.

Comments on the paper

4 comments
Area_Chair_Wj2zOpenReview ↗

From AC.

Dear Authors,

Given the deadline on 18 November, please submit your responses to the reviews to facilitate a productive discussion with the reviewers.

Thanks, AC

Authors: the author-reviewer debate about the paper is an integral and important part of the reviewing process. If you can, please submit your responses to the reviews.

AuthorsOpenReview ↗

Responses to Chair

Dear Area Chair and reviewers,

Thank you for the extremely useful feedback, and sincere apologies for the late reply. The reviews prompted new experiments to address the feedback, and we wanted them finished before responding. A revised PDF has been uploaded together with this response.

The main changes in the revised manuscript are:

  • Causal tests (DmAd W4, Q5). Activation steering in three environments provides evidence that the policy acts on the decoded belief coordinates its task requires (Sections 3.4, 4.2 and 5.2).
  • Probing beyond what behaviour implies (DmAd W2, Q1). We now probe within greedy-action classes and against each network's own initialization on identical histories (Appendix G). We no longer claim that belief geometry "emerges"; we claim that training refines it inside decision regions, along the coordinates the task requires.
  • Revised ladder environment (DmAd W2, Q2). The within-class analysis showed that the submitted ladder made little use of memory: policies using only the current token obtained essentially the same return as the transformer policies. We therefore changed the environment's parameters. It remains a three-state hidden Markov process with one rewarded state and tilt actions, but its latent dynamics are more persistent and its observations less informative, so that acting well requires integrating evidence over the history. The ladder experiments were rerun in this environment, and the Reward Relevance experiment is built from it, so their figures and numbers differ from the submitted version (Section 3.3; Appendix C.1).
  • Behavioural applications and a standard POMDP benchmark. We add transfer to a changed reward through externally guided steering without policy-weight updates (Section 4.2), and RockSample experiments that diagnose neglected belief coordinates and selectively redirect rock collection (Section 5).
  • Clearer positioning and scope (DmAd W1, W5; MDW3 W2). The hypotheses are restated (H1: adaptive beliefs; H2: causal role), the title is changed, the environment and probe definitions are in the main text, and the introduction and Section 6.1 state what we add to Lambrechts et al. (2022) and Hennig et al. (2023).

In summary, our contribution concerns task-dependent belief geometry and its behavioural consequences in the studied settings. The proposed learning-dependent separation principle remains an interpretation motivating these experiments, rather than a general theorem about what reward optimization must learn. We appreciate the reviewers' suggestions and feedback, which allowed us to substantially strengthen the paper.

Note that the reviews cite the submitted version. Old Section 4 (ladder) is now Section 3.3, old Section 5 (reward relevance) is now Section 4, and Section 5 is now RockSample. Our replies use the new numbers.

Point-by-point replies follow under each review.

We are extremely sorry for only submitting our response towards the end of the period. As we said above, the feedback motivated us to include various changes, and we wanted to finish the new experiments before responding. Apologies for the inconvenience.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.