acceptodds
Under review as a conference paper at ICLR 2027

The Belief-Action Gap in Language Model Agents

Abstract

Language model (LM) agents can fail for different reasons, requiring different explanations and interventions. We study how state beliefs, action values, and action preferences interact as LM agents reason and commit to decisions in text-rendered grid worlds. We estimate task-relevant beliefs using report-based and representation-based probes, and action values using inverse reinforcement learning. We find that task-relevant state and value information is often represented more reliably than it is reflected in behaviour. Tracking action preferences throughout reasoning further shows that decisions sharpen around a commitment point and reported belief uncertainty decreases near commitment, although individual belief changes do not consistently explain shifts in action preference. Finally, forced actions follow reasoning prefixes from other decisions when transplanted, whereas the tested reasoning-window activation edits provide limited control over the action. Together, these results suggest that some failures arise not from missing task-relevant information, but from how available information is translated into action. Our measurements distinguish information availability from belief-action consistency without by themselves identifying the cause of each failure. Our code is available at: https://anonymous.4open.science/r/belief-action-gap-F745/

Primary area:
interpretability and explainable AI
Keywords:
decision-making, interpretability, belief elicitation, value inference, machine cognition

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_VBbLOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to disentangle an LLM agent's belief, preference and action.

The authors tackle this problem with inverse RL (to estimate the value functions) and linear probes (to decode model's internal belief of the world).

I would say that the main contribution of this paper is the creative use of the above tools to understand the "belief-action gap" in an LLM agent. The experiment design to probe representations before and after reasoning trace is also thoughtful and reveals that reasoning help closes the belief-action gap.

Experiments are done in a grid world environment, so it lacks diversity and realism, but it is a good controlled environment to study the problem formulated here. I don't find it a major concern.

I do have some concerns regarding the validity of the approach. I will detail them in later sections.

Critical strengths

The experiments are well-designed. First block of experiment measure the functions, whereas second block understands CoT, and the last block explores intervention and seeks for causal proof of previous findings. Interesting empirical findings are shown, demonstrating that <reasoning> helps <aligning action with belief and value>, but not <forming a more accurate belief>.

I also like that the paper studies the effect of CoT reasoning (introducing a 4th variable other than belief, value, and action), by injecting probing questions about next action in the middle of the CoT traces. The result that model forms their final action preference in the middle of the reasoning trace is surprising (though aligned with other relevant work in LLM response planning).

Critical weaknesses

One concern is the validity of the approach.

The Inverse RL model uses the state representations and the observed action to infer preferences, but the model may actually be acting on an inaccurate belief about the state! You are not decoding the belief from the model in this step, so you won't know the model's belief when fitting the preference model. If so, an action mistake caused by an incorrect belief could be interpreted as a preference error. This makes it unclear how well the framework can actually disentangle these two sources of failure.

Another concern is the lack of reference for closely related work. I don't know if there's more, but here are two missing papers that are very relevant: [1] also decodes beliefs from the representations of LLMs and intervene on them, and [2] (which is from ICLR 2026) discusses a more realistic belief-action gap where the model can infer the belief but fail to infer the action led by the belief.

Moreover, a minor weakness is the presentation. I find a lot of terms in the paper not well-defined, especially not explained in their first appearances. E.g., I have to read more and do in-context learning to understand "state preference" is the value function (real preference), and "action preference" is the potential action distribution of the agent. "Pre-reasoning" and "post-reasoning" also appears early in 2.3 but not explained until later sections

[1] Zhu, W., Zhang, Z., & Wang, Y. (2024). Language models represent beliefs of self and others. arXiv preprint arXiv:2402.18496.

[2] Gu, Y., Tafjord, O., Kim, H., Moore, J., Le Bras, R., Clark, P., & Choi, Y. (2026, April). Simpletom: Exposing the gap between explicit tom inference and implicit tom application in llms. In International Conference on Learning Representations (Vol. 2026, pp. 50252-50297).

Questions for authors

In the wall-hit example, the model has the correct belief about the world but chooses the incorrect action. You suggest that this is due to the belief-action gap. I don't quite get it because if the model looks forward for one single step in their reasoning trace, they would know that there is a wall. So I was wondering if this could be due to the model's incorrect belief of the world transition function (which is not measured in this work)?

3 comments

We thank the reviewer for these constructive comments. We address the methodological concern, the suggested references, and the terminology below.

Belief--preference confounding in IRL. We agree that an agent may select an action based on an incorrect belief about the environment state. Since our one-step IRL model is fitted using the observed environment states and the agent's actions, the inferred state values may partially reflect the consequences of inaccurate beliefs rather than preferences alone. This is an important limitation of inferring preferences from observed behaviour.

Our framework does not assume that the inferred state values uniquely recover the agent's internal preferences. Rather, the IRL model provides an operational estimate of the state preferences expressed through the agent's behaviour. We then combine these inferred values with independently estimated belief distributions to quantify the consistency between represented state information and selected actions.

In particular, our decomposition distinguishes the expected loss associated with the emitted action under the estimated belief from the expected value of resolving state uncertainty. These quantities are well-defined under the fitted belief and value models, but we do not claim that they uniquely identify the underlying causes of individual action errors.

Importantly, our results are not restricted to the observations used to fit the estimators. As reported in Table 1, the belief--action gap decreases from 0.316 to 0.199 on the jointly held-out subset of 762 decisions, whose observations were excluded from both probe and IRL training. This supports the generalization of the measured relationship, although it does not by itself eliminate belief--preference confounding.

An alternative would be to fit the IRL model directly using the estimated belief distributions rather than the true environment states. However, this would make preference inference dependent on the accuracy and calibration of the belief probes, potentially introducing additional estimation errors.

Relationship to prior work. We thank the reviewer for pointing out these relevant references, which we will include into the revised manuscript.

Zhu et al. (2024) investigate representations of agents' beliefs in language models and demonstrate that belief-related information can be decoded from internal activations and influenced through activation interventions. This work is related to the representational and causal aspects of our study. Our focus, however, is on the relationship between an agent's estimated environment-state beliefs, its inferred state preferences, and its own action selection. In particular, we introduce a quantitative decision-theoretic diagnostic that combines activation-derived beliefs with IRL-inferred values.

Gu et al. (2026) demonstrate a gap between explicit Theory-of-Mind inference and the application of that information in predicting or evaluating behaviour. This finding is conceptually related to our motivating question of whether represented information is consistently reflected in decisions. Our setting differs in that we study an agent's own environment-state beliefs and actions rather than its predictions about another person's behaviour. We additionally quantify action consistency through expected value differences under estimated beliefs and inferred preferences, and investigate how this relationship evolves during reasoning.

Clarity of terminology. We agree that several terms should be defined more clearly when first introduced. We will revise the presentation to distinguish state preferences, referring to the scalar state values inferred through IRL, from action preferences, referring to the model's elicited probability distribution over candidate actions. We will also define pre-reasoning and post-reasoning at their first appearance, clarifying that they refer to measurements obtained before and after the model generates its reasoning, respectively.

Question 1. We thank the reviewer for this observation. We agree that an agent may correctly identify its current environment state while incorrectly predicting the consequences of an action. Such an error could contribute to the observed wall-hit failures. Our analysis in Appendix B.1 examines three wall-hit decisions in which the model selects a direction that is blocked by a wall. In all three cases, the model correctly reports that the selected direction is blocked. These examples demonstrate that the local state information needed to avoid the error is explicitly recoverable, even when the agent ultimately selects the blocked action. Importantly, we do not interpret these results as establishing that the model necessarily used this information during its original decision. As discussed in Appendix A.2, report-based beliefs are elicited through separate forward passes and may recover information that was not used when selecting the action.

Thanks for the response! The rebuttal addresses most of my concerns. I like the design of the experiments and the methodology, though more limitations need to be acknowledged, e.g., the method cannot reliably distinguish between belief error or preference error. It would be desirable if you could derive a more accurate method in disentangling different error sources. And yes please discuss these relevant works.

I have increased my rating because I think this work is novel and interesting, but my concern about the approach remains, so I lowered my confidence at the same time. It really depends on different people's internal value functions to position this paper.

We thank the reviewer for their thoughtful follow-up and for reconsidering our work. We appreciate the positive assessment of our experimental design and methodology.

We appreciate the reviewer's constructive feedback, which has helped us clarify the scope and limitations of our approach.

Reviewer_EYikOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the gap between task-relevant information represented within language model agents and their actual action choices. In a controlled text-rendered grid-world environment, the authors estimate the agents' state beliefs using internal activation probes and explicit state reports, infer state preferences through one-step inverse reinforcement learning (IRL), and introduce a metric based on expected local regret to quantify belief–action consistency.

Critical strengths

The paper combines state belief estimation, preference inference, and decision-theoretic metrics into a unified framework for analyzing agent behavior. This framework provides multiple perspectives for understanding why language model agents make suboptimal decisions and offers a potentially useful approach to studying their decision-making mechanisms.

Critical weaknesses
  1. The reliability of the belief–action gap metric is insufficiently validated. The paper's main conclusions rely on two indirect estimates: state beliefs inferred from internal activation probes and state values inferred from the agent's own actions. First, the belief estimates are not sufficiently accurate. The post-reasoning position probe achieves only 13.1% top-1 accuracy, raising concerns about whether the estimated beliefs accurately reflect the information represented by the model. Similarly, the agent-supervised IRL models achieve accuracies of only 29.2% and 45.2% before and after reasoning, respectively, raising questions about the reliability of the inferred state preferences. Second, the authors fit separate value models for the pre-reasoning and post-reasoning conditions, each supervised by the corresponding actions produced by the agent. Consequently, the observed reduction in the belief–action gap may reflect differences in belief uncertainty, value-model fitting, or value scales rather than a genuine improvement in belief–action consistency.
  2. Limited experimental generalizability and insufficient differentiation from prior work. The main experiments are conducted using a single fixed grid-world layout and one language model, making it difficult to determine whether the observed phenomena generalize to other models or more complex environments. Moreover, several closely related studies have already investigated similar questions. Arghal et al. (2026) [1] examined the relationship between internal state representations, reasoning, and agent behavior in grid-world environments, while Boppana et al. (2026) investigated early answer commitment during language model reasoning. The paper does not sufficiently establish which unique and important insights its proposed framework provides beyond these existing studies.

[1]A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

[2]Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

Questions for authors
  1. The post-reasoning position probe achieves only 13.1% top-1 accuracy on held-out states. How sensitive are the paper's main conclusions to probe prediction errors and differences in probability calibration?
  2. Can the authors conduct more directly comparable causal intervention experiments, such as reasoning-text interventions that modify only state information without explicitly suggesting an action, and action-targeted activation interventions with effective positive controls? Such experiments would help distinguish the causal influence of represented information on action selection.
5 comments
AuthorsOpenReview ↗

Weakness 1 Response

We thank the reviewer for raising these concerns regardingcthe reliability and comparability of our belief--action gap estimates.

Reliability of belief estimates. While exact location prediction is relatively low, top-1 accuracy alone does not fully characterize the estimated belief distributions used in our analysis. As reported in Appendix C.1, the predicted location lies within Manhattan distance one of the true location in 95.5% of pre-reasoning and 60.8% of post-reasoning cases, and within distance two in 99.0% and 83.5%, respectively. Moreover, the belief--action gap is computed using the full predicted state distribution rather than the top-1 prediction. The probability-mass analysis further shows that these distributions retain substantial spatial information, with 93.1% and 59.2% of probability mass within distance one before and after reasoning, respectively. We acknowledge that these estimates are imperfect, particularly after reasoning, but they retain meaningful spatial information beyond exact location classification.

Reliability of inferred state values. Our one-step IRL model estimates subjective state preferences from the agent's observed actions rather than attempting to recover its true internal value function. The relatively modest action-prediction accuracies (29.2% pre-reasoning and 45.2% post-reasoning) should also be considered in the context of stochastic action generation and the restricted linear value model. As an additional control, Appendix C.2 demonstrates that the same modelling approach can recover planner-optimal action structure substantially more accurately when trained with optimal-action labels. This suggests that the difficulty of predicting the agent's actions is not solely attributable to the expressiveness of the representations or value model, although it does not eliminate uncertainty in the inferred preferences.

Separate models and held-out evaluation. We intentionally fit separate pre-reasoning and post-reasoning value models because the two conditions use different activation-derived representations and corresponding action labels. This allows us to estimate condition-specific preference structures without assuming that the inferred preferences remain unchanged throughout reasoning. Importantly, the reduction in the belief--action gap is also observed on the 762-decision held-out subset in Table 1, whose observations were excluded from both probe and IRL training. On this subset, RR decreases from 0.316 to 0.199, demonstrating that the result is not restricted to observations used to fit the estimators.

Belief uncertainty and value-scale comparability. We agree that changes in decoded belief uncertainty and differences in the numerical scales of the inferred value functions may contribute to the observed decrease in RR. To investigate this, we conducted the random-action normalization experiment also suggested by other reviewers. Specifically, we normalize the belief--action gap by the expected regret of a uniformly random action, RrandR _{\mathrm{rand}}, under the corresponding estimated belief and inferred value function:

Rrand,t,p=max⁡a′∈AEs∼q^t,p[gθp(TG(s,a′))]−1∣A∣∑a∈AEs∼q^t,p[gθp(TG(s,a))].R _{\mathrm{rand},t,p} = \max _{a'\in\mathcal A} \mathbb{E} _{s\sim\hat q _{t,p}} \left[g _{\theta_p}(T_G(s,a'))\right] - \frac{1}{|\mathcal A|} \sum _{a\in\mathcal A} \mathbb{E} _{s\sim\hat q _{t,p}} \left[g _{\theta_p}(T_G(s,a))\right].

We then compute the normalized gap R~t,p=Rt,p/Rrand,t,p\widetilde R _{t,p}=R _{t,p}/R _{\mathrm{rand},t,p} and report the mean of the per-decision ratios.

The normalized gap decreases from 0.62460.6246 before reasoning to 0.48980.4898 after reasoning across all 3,1943{,}194 decisions. The decrease also persists on the 2,4652{,}465 decisions where all probes are correct in both conditions (0.66600.6660 to 0.51050.5105), and on the 762762 jointly held-out decisions (0.49390.4939 to 0.42710.4271). No decisions were excluded due to near-zero normalization denominators.

Importantly, the mean random-action regret does not decrease after reasoning, increasing from 0.83230.8323 to 0.86030.8603 across all decisions and from 0.69300.6930 to 0.76470.7647 on the held-out subset. Furthermore, the normalized gap is invariant to positive multiplicative rescaling of the inferred state values. These results indicate that the observed reduction in RR cannot be explained solely by differences in the overall value scales or a general compression of expected action-value differences.

While this analysis does not completely isolate the effects of belief uncertainty or differences in the fitted preference structures, it provides additional evidence that the reduction in the belief--action gap persists beyond these simple scaling effects. We will incorporate these results into the revised manuscript and clarify the scope of their interpretation.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 1

Weakness 2. We thank the reviewer for highlighting the need to clarify our contributions relative to prior work. While we adopt the grid-world setting of Arghal et al. (2026), our work addresses a distinct research question. Arghal et al. investigate goal-directedness through behavioural and representational evaluations, including the consistency of actions with decoded cognitive maps. In contrast, we investigate how estimated state beliefs and inferred preferences relate to the actions selected by language model agents. To study this relationship, we introduce a quantitative framework combining probabilistic state beliefs with subjective state values inferred through one-step IRL. Our belief–action decomposition separates the expected loss associated with the emitted action under the estimated belief from the expected value of perfect state information. We further investigate how action preferences evolve during reasoning, how their revisions relate to reported state beliefs, and how reasoning-text and activation interventions influence action selection.

Similarly, Boppana et al. (2026) investigate when a model's eventual answer becomes recoverable during reasoning. While our results also identify early action commitment, our focus is on how commitment relates to task-state beliefs, action optimality, and causal influences on action selection in an interactive environment. Our contribution therefore lies in jointly analysing information availability, inferred preferences, decision formation, and behavioural control, rather than in the observation of early commitment alone.

Question 1. We thank the reviewer for raising this important question. While the post-reasoning position probe achieves 13.1% top-1 accuracy on held-out states, exact classification accuracy alone does not fully characterize the estimated belief distributions used in our analysis.

As reported in Appendix C.1, the predicted location lies within Manhattan distance one of the true position in 60.8% of post-reasoning cases and within distance two in 83.5% of cases. The corresponding percentages are 95.5% and 99.0% before reasoning. Furthermore, the predicted distributions assign an average of 59.2% and 81.5% probability mass within Manhattan distances one and two, respectively, after reasoning. These results indicate that the probes retain substantial spatial information even when their exact location predictions are incorrect.

Importantly, the belief--action gap is computed using the full estimated belief distribution q^t,p(s)\hat q _{t,p}(s) rather than its most probable state. Nevertheless, we agree that inaccuracies in the estimated probabilities, including differences in calibration between the pre-reasoning and post-reasoning probes, may influence the resulting gap measurements. To assess this sensitivity, we conducted an additional analysis using temperature-scaled belief distributions:

q^t,p(τ)(s)=q^t,p(s)1/τ∑s′∈SGq^t,p(s′)1/τ,τ>0.\hat q _{t,p}^{(\tau)}(s) = \frac{ \hat q _{t,p}(s)^{1/\tau} }{ \sum _{s'\in S_G} \hat q _{t,p}(s')^{1/\tau} }, \qquad \tau>0.

We evaluated τ∈{0.5,0.75,1,1.5,2}\tau\in\{0.5,0.75,1,1.5,2\}, where τ=1\tau=1 corresponds to the original estimated belief, τ<1\tau<1 produces sharper distributions, and τ>1\tau>1 produces more diffuse distributions. This transformation preserves the ranking of state probabilities while varying their concentration. For each temperature, we recomputed the belief--action gap Rt,pR _{t,p}, the expected value of perfect state information It,pI _{t,p}, and belief entropy Ht,pH _{t,p}, keeping the corresponding inferred value models fixed.

The results show that the belief--action gap decreases from pre- to post-reasoning at every evaluated temperature across all three subsets. Across all 3,1943{,}194 decisions, RR decreases from 0.5422 to 0.3523 at τ=0.5\tau=0.5, from 0.4673 to 0.2677 at τ=1\tau=1, and from 0.3276 to 0.2155 at τ=2\tau=2. The same pattern holds for the 2,4652{,}465 decisions where all probes are correct in both conditions.

Importantly, the reduction also persists on the 762762-decision held-out subset. At τ=0.5\tau=0.5, RR decreases from 0.44000.4400 to 0.25370.2537; at τ=1\tau=1, from 0.31570.3157 to 0.19870.1987; and at τ=2\tau=2, from 0.23460.2346 to 0.18840.1884. The intermediate temperatures yield the same qualitative result.

We also find that the expected value of perfect state information and decoded-belief entropy generally increase after reasoning across the evaluated temperatures. Thus, the reduction in RR persists under substantial changes in belief concentration, even as the estimated beliefs remain more uncertain after reasoning.

These results show that our main finding remains qualitatively robust across the tested variations in belief sharpness, although the magnitude of the gap reduction depends on temperature. However, temperature scaling tests sensitivity rather than probability calibration. It does not correct potential errors in the probes' state rankings or confirm that their predicted probabilities reflect the model's internal beliefs.

Thank you for the detailed rebuttal and additional experiments. I appreciate the random-action normalization and belief-temperature sensitivity analyses, which help address my concerns about value-scale differences and belief uncertainty. The clarification of the contributions relative to prior work is also helpful.

However, three concerns remain:

  1. The agent-supervised IRL models still have relatively low predictive accuracy. Although the new normalization experiment addresses simple scaling effects, the use of separate pre- and post-reasoning value models and potential belief–preference confounding leave uncertainty about whether the observed gap reduction reflects improved belief–action consistency rather than differences in the fitted preference models.

  2. The additional model experiments have not yet been reported, so it remains unclear whether the key findings generalize beyond a single model and fixed grid-world environment.

  3. The authors clarify the motivation and limitations of the existing causal interventions, but the requested more comparable text and activation intervention controls have not been provided.

Overall, the rebuttal provides meaningful improvements, and I appreciate the authors' efforts. However, these remaining concerns, particularly regarding the reliability of the core metric, prevent me from changing my assessment. I will therefore maintain my original score.

Reviewer_HNreOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the question of how state representations, state values, and action preferences affect the decisions of LM-based agents and in particular, how these components can be used to understand cases where agents fail to make optimal decisions. The work makes several contributions, including (i) a decomposition of expected local regret (a quantitative framing of belief-action consistency) into two terms (how much value is lost by the chosen action vs the best action according to the model's state belief distribution and a quantity that measures the theoretical gain of perfect state information), (ii) the identification of a "commitment point" during reasoning, after which the preferred action stabilizes, (iii) evidence that interventions directly to reasoning text can change action preferences, and (iv) general supporting evidence in a simple, principled setting of a belief-action gap (that is, that models can encode state information reasonably strong while still selecting suboptimal decisions). They validate their claims experimentally through a very simple, full-information, deterministic grid-world setting.

Critical strengths
  • I find the proposed methods of estimating representation-based belief (q^t,p(s)\hat{q}_{t, p}(s)), value estimates (g(ϕ(s))g(\phi(s))), and expected local regret (Δ\Delta) to be quite reasonable and generally well motivated. In particular, I find the decomposition of expected local regret to be quite insightful, composed of terms RR which expresses how much inferred value is lost under the chosen action vs. best perceived and II which captures how valuable perfect knowledge of the state is. I think the community in general can benefit from this perspective and formal quantification of belief-action consistency.
  • The finding that models "commit" to an action over the course of reasoning and that this stabilizes is interesting and supported clearly. This insight may inspire several potential future works, e.g. if models commit to the action with roughly 50% of the reasoning trace, one could explore the early stopping of reasoning in an effort to save compute resources.
  • The paper goes beyond correlational analysis and displays several causal results. In particular it is an interesting insight to the community that a single sentence can override confident decisions: the forced top action switched to the injected action in 62.5% of cuts with c≥0.95c \geq 0.95 (and 100% with c<0.7c<0.7). Taken in conjunction with the commitment finding, this may suggest that commitment is stable under a model's own completions but fragile to new text.
Critical weaknesses

Comments that affect my decision (I apologize if line numbers are off by 1). I am willing to consider increasing my score if the following could be addressed.

  • In lines 243-244, it is claimed "After reasoning, the emitted action is therefore more consistent with the decoded belief...". Table 1 indeed provides initial evidence for this in that RR decreases from pre- to post- in each of the three settings, but I am not sufficiently convinced that this is not all explained by the increase in decoded belief entropy HH. What I mean by this is the following: the decrease in R from pre- to post- could be explained by the trend in HH: higher entropy means a flatter belief distribution, which may pull the actions' expected values closer together, in turn decreasing RR. One simple experiment that I could imagine addressing this is the following: compute Rrand=max⁡a′Es∼q^t,p[gθp(TG(s,a′))]−1∣A∣∑aEs∼q^t,p[gθp(TG(s,a))],R_{\mathrm{rand}} = \max_{a'} \mathbb{E}_{s \sim \hat{q}_{t,p}} [ g_{\theta_p}(T_G(s,a')) ] - \frac{1}{|A|} \sum_{a} \mathbb{E}_{s \sim \hat{q}_{t,p}} [ g_{\theta_p}(T_G(s,a)) ], then measure R/RrandR/R_\mathrm{rand} between pre- and post-reasoning settings. If the decrease in RR were to due increased entropy alone, R/RrandR/R_\mathrm{rand} would remain roughly unchanged; on the other hand, if it decreased pre- to post-, then the emitted action becomes more consistent with decoded belief beyond what the increase in entropy explains.
  • I found the motivation of Section 5.2 unclear. The section states that it tests whether action choice can be controlled through a reasoning-window representation from which action decisiveness (measured via inverse normalized entropy) is strongly decodable. The chosen site of editing is justified via decodability of decisiveness/confidence, but the experiment then measures whether edits change which action is chosen. These are different quantities: two prefixes favoring different actions with equal confidence have identical decisiveness as measured. Decodability of decisiveness at this site does not necessarily imply that action choice is represented there in editable form, so the negative result is perhaps unsurprising. I list this as a weakness, but it is possible that I am misunderstanding the motivation of the experiment or the experiment itself. I wonder if the object of interest is the chosen action, could one instead make edits to regions that have high decodability for action identity? Or if the object of interest was decisiveness, could we measure instead the forced action entropy after the edit?
  • To my understanding, the experiments use a single model (gpt-oss-20b) in the simple grid-world environment. The controlled environment is well justified and understandable (and also appropriately mentioned as a limitation), but it would strengthen the paper to replicate key findings (e.g. commitment dynamics or the expected local regret decomposition) on at least one other additional model, or at the very least discuss whether the findings are expected to generalize and to what extent.

Additional comments (minor)

  • Line 125: can you briefly clarify what is meant by token region?
  • Lines 131-136: the appendix provides clear explanations of report-based estimates, but you may consider including a concrete example somewhere here so that the reader has a clearer idea of what the report-based probing does.
  • Line 196: gpt-oss-20b is mentioned here without citation; consider adding a citation.
  • Line 287: it is not immediately obvious what a forced-answer is; consider briefly explaining this.
  • Figure 3 caption: consider changing "left." -> "Left:" and similarly "right." -> "Right:".
Questions for authors
  • Lines 143-144: It is stated that the state-value function is linear in this representation. Is this a property of the setting? This is not obvious to me, and I believe it is actually a modeling assumption made in two papers that are cited in the same paragraph.
  • Line 205: Why were layers 7, 15, and 23 chosen? Did you try any other layers?
  • Figure 2 caption: I inferred that the left figure is pre-reasoning and right figure is post-reasoning. Is that correct or am I misunderstanding this? If so, would you consider labelling the panels?
  • Figure 3: What is the split of data samples between up/down/left/right used in these results? The uniform prior baseline is mentioned here, but this may differ from say a class-frequency predictor if the action classes are not balanced equally.
  • In Section 4.3 and Appendix D.4, regarding the clustering used, what was the reason for k=12? Did you try other values of k?
  • Similarly to the above point on clustering, could you elaborate on how clusters were "associated post hoc with solving and action consolidation"? Were labels annotated by a human or LLM?
  • (future work question) From the first experiment in section 5.1 regarding transplanting the reasoning of B into A, I wonder what happens if you allow the model to continue to reason after the transplant? Does it continue to follow B's reasoning or does it show any signs of reverting back to A's? Did you try any such experiments?
6 comments
AuthorsOpenReview ↗

Regarding Weaknesses

We thank the reviewer for their careful reading of our manuscript and their constructive feedback. Below, we address the comments point by point.

Weakness 1. We thank the reviewer for suggesting this additional analysis. We agree that increasing belief uncertainty could reduce the differences between expected action values and thereby contribute to the observed decrease in RR.

To investigate this possibility, we conducted the proposed experiment, normalizing the belief--action gap by the expected regret of a uniformly random action:

R~t,p=Rt,pRrand,t,p,\widetilde R _{t,p} = \frac{R _{t,p}}{R _{\mathrm{rand},t,p}},

We report the mean of the per-decision normalized regrets. Across all 3,1943{,}194 decisions, the normalized gap decreases from 0.62460.6246 before reasoning to 0.48980.4898 after reasoning. The decrease also persists on the 2,4652{,}465 decisions where all probes are correct in both conditions (0.66600.6660 to 0.51050.5105) and on the 762762 jointly held-out decisions (0.49390.4939 to 0.42710.4271).

Importantly, the mean random-action regret does not decrease after reasoning, changing from 0.83230.8323 to 0.86030.8603 across all decisions and from 0.69300.6930 to 0.76470.7647 on the held-out subset. Thus, the reduction in RR is not simply explained by a general compression of expected action-value differences.

Our results show that the normalized gap decreases consistently across all three evaluation subsets, providing additional evidence that the observed improvement in belief--action consistency is not merely an artifact of shrinking action-value differences. We will include these results in the revised manuscript.

Weakness 2. We thank the reviewer for this comment and agree that action decisiveness and action identity are distinct quantities. Our motivation in Section 5.2 was to investigate whether a reasoning-window representation strongly associated with action commitment could also serve as an effective intervention site for controlling action selection.

We first establish that action decisiveness is decodable from this representation (held-out Spearman ρ=0.771±0.009\rho=0.771\pm0.009). We then test whether modifying the same representation can redirect the model's preferred action. This is an exploratory causal test of whether a representation associated with decision formation also provides control over the resulting action, rather than an assumption that decisiveness and action identity are equivalent.

Our results show that targeted rank-1 steering produces the intended action in only 3.6% of cases, matching the random-direction control. In contrast, full-window activation transplantation redirects 20.8% of cases towards the donor-associated action. Together, these results indicate that action selection can be influenced through this representation, but that the effectiveness of control depends on the intervention method.

We will clarify this motivation in Section 5.2 and emphasize that our conclusions concern the specific representations and intervention methods evaluated.

Weakness 3. We are right now conducting the additional model experimentation and will also add its results in the latest version, and will be sharing here as well if the experiments finalise before the rebuttal period.

Minor Weaknesses.

  • For line 125, the token regions are three final prompt tokens before reasoning and the three final reasoning tokens immediately before action generation. We discuss these in Appendix A1 and will make it clear in the main text.

  • For line 287 a forced answer is elicited by appending a fixed answer-forcing suffix to the environment prompt, optionally followed by a reasoning prefix, and evaluating the logits of the four candidate actions to obtain the model's action preferences without allowing further reasoning. We discuss these in Appendix D1 and will make it clear in the main text.

  • For the remaining points, we will follow the reviewers suggestions in the manuscript.

AuthorsOpenReview ↗

Regarding Questions

We thank the reviewer again for their careful reading of our manuscript and their constructive feedback. Below, we address the questions.

Question 1. The reviewer is right to point out this. The linearity of the inferred state-value function is indeed a modelling assumption rather than an inherent property of the environment or language model. We adopt a linear parameterization over activation-derived state representations, gθp(s)=θp⊤ϕp(s)g _{\theta_p}(s)=\theta_p^\top\phi_p(s), to estimate state preferences from observed actions. We will revise the wording in Section 2.3 to make this distinction explicit.

Question 2. Layers 7, 15, and 23 were selected as representative early, intermediate, and late layers of gpt-oss-20b, allowing us to examine the recoverability of task-relevant information at different network depths. This choice is motivated by prior probing work studying how information is distributed across transformer layers Arghal et al. (2026) [1], rather than by an assumption that these particular layers are optimal. We will clarify this motivation in the manuscript.

Question 3. The reviewer is correct that the left panel shows the pre-reasoning condition and the right panel shows the post-reasoning condition. We will clarify this in the caption of Figure 2 and label the panels accordingly.

Question 4. We thank the reviewer for highlighting the importance of the empirical action distribution. The 1,2761{,}276 decisions in Figure 3 consist of 472 UP (36.99%), 172 DOWN (13.48%), 247 LEFT (19.36%), and 385 RIGHT (30.17%) actions. We agree that the uniform baseline of 0.250.25 does not account for this imbalance. Using the empirical action frequencies, the corresponding baseline is 0.28350.2835. This baseline is slightly above 0.250.25, but remains below the approximately 0.350.35 probability observed early in reasoning and well below the approximately 0.920.92 reached near the end. Furthermore, Figure 3 (right) shows that the probability of the eventual action increases throughout reasoning for all four actions, despite differences in their initial probabilities. This indicates that the trend is not driven solely by the most frequent actions. We will clarify the baseline interpretation and report the empirical action distribution and frequency baseline in the revised manuscript.

Question 5. We experimented with several cluster counts, k∈{4,8,12,16}k\in\{4,8,12,16\}, and selected k=12k=12 based on the interpretability of the resulting clusters. With smaller values of kk, clusters tended to combine representations associated with different reasoning functions, making their interpretation ambiguous. With larger values, clusters became more fragmented, often producing groups without a clear interpretation or subdividing patterns already captured at k=12k=12. We therefore selected k=12k=12 as a practical balance between representational granularity and interpretability. We will clarify this selection procedure in Appendix D.4.

Question 6. The clustering was performed entirely unsupervised, with semantic interpretations assigned only after clustering. To interpret the clusters, we compared their assignments with independently generated sentence-level reasoning-function annotations and inspected representative sentences from each cluster. The reasoning-function annotations were produced by gpt-oss-20b using a fixed rubric adapted from Bogdan et al. (2025), and evaluated against 152 blindly author-labelled sentences.

The associations with solving and action consolidation are therefore post hoc interpretations rather than predefined cluster labels. We also note that the correspondence between clusters and semantic annotations is modest (normalised mutual information of 0.131), so we use numerical cluster identities in the quantitative analyses rather than treating clusters as distinct semantic categories. We will clarify this procedure in Section 4.3 and Appendix D.4.

Question 7. The reasoning transplantation experiment in Section 5.1 aims to measure the immediate causal effect of transplanted reasoning on the model's action preference. We therefore measure the forced-action distribution directly after transplantation, without further reasoning. Across 396 mismatched prompt and reasoning pairs, the model assigns substantially higher probability to the action supported by the transplanted reasoning (0.5980.598 to 0.6730.673) than to the action supported by the original prompt (0.1290.129 to 0.1650.165). This shows that transplanted reasoning can redirect the model's immediate action preference, even when it conflicts with the original prompt. We have not tested whether this effect persists when the model continues reasoning. Such an experiment would reveal whether the model maintains the transplanted preference or returns to its original preference after further reasoning. We agree this is an interesting direction for future work.

[1] A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

Thank you very much for the response! I especially appreciate that you conducted the random action normalization experiment (W1) as well as the empirical action distribution baseline (Q4). Thank you also for clarifying the motivation in Section 5.2 (W2) and answering my many questions in detail.

My remaining concern is generalization beyond gpt-oss-20b. It would be ideal for me to have the chance to see the generalization of your claims to the additional model before I raise my score. Do you have a sense of whether the experiments will be finalized before the Author/Reviewer discussion period ends? Should the key findings replicate on another model, I would be inclined to raise my score.

Reviewer_aLoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The work is related to study of LLMs translation of belief and preferences into actions. The study is conducted over a controlled synthetic grid world with deterministic state transition function. The work discuss that task-relevant information is available but is not consistently used for action selection by the models. Authors have used existing works of Inverse reinforcement learning (IRL) to motivate the modeling part and quantified the expected local regret. The study is presented for a single model of gpt oss 20B.

Critical strengths

The idea of quantifying belief-action gap is interesting and is increasing becoming relevant in the community as the use of agentic models is gaining momentum. The reviewer also appreciate the mentions of weak actions in introduction as good motivation, and finally posing the problem as inverse RL and quantifying the regret.

Critical weaknesses

The main concern with the work is limited theoretical and experimental analysis. I list them below:

  1. Only using grid world to make general claims regarding belief-action gap is not a good result. Additionally, since only a single model (gpt oss 20B) is used, I would say that there is significant doubt in reviewers mind, that such analysis would hold for different models/datasets. (authors also ack the dataset issue in line 464-465)
  2. The authors must also focus on making the analytical results better. For example, in eq (4)-(5), its straight-forward that eq(4) R_t,p is always non-negative, but what about eq(5) I_t,p? I think there must be some role of Jensen's equality here which could help in proving it positive. O/w its difficult to interpret the break of \Delta_t,p into 2 quantities
  3. In the interpretation part, line 183-185, how did authors mention "remaining state uncertainty..." in eq(5)? I am not seeing any connection here
  4. line 142-143, what are these separate models? atleast a ref to appendix would be helpful
  5. in eq.(1), the gibbs distr (or boltzman distr as written by authors) must be motivated. I see that the expressions are mostly motivated from ref (Ziebart 2008), but a clear mention of using max entropy and then this distributions would help reader to make a connection
Questions for authors

pls see the weakness section.

2 comments

We thank the reviewer for their careful reading of our manuscript and their constructive feedback. Below, we address the comments point by point.

Question 1. We use the grid world as a controlled abstraction of sequential, goal-directed decision making, in which an agent must represent its environment and select actions on the basis of that representation. We do not present the observed belief–action gap as a universal property of large language models. Rather, our results demonstrate that such a gap can occur and can be measured systematically. We provide a framework and an experimental approach for analysing it. We hope that future work will extend this analysis to a broader range of environments and models.

We are right now conducting the additional model experimentation and will also add its results in the latest version, and will be sharing here as well if the experiments finalise before the rebuttal deadline.

Question 2. The comment is correct in mentioning Jensen's inequality, however the positivity of these equations is straightforward.

The non-negativity of It,pI_{t,p} follows directly from the fact that, for any fixed action aa,

max⁡a′∈AXa′(s)≥Xa(s),\max_{a'\in\mathcal A} X_{a'}(s) \geq X_a(s),

where Xa(s)=gθp(TG(s,a))X_a(s)=g_{\theta_p}(T_G(s,a)).

Taking expectations and then maximizing over aa gives

Es[max⁡aXa(s)]≥max⁡aEs[Xa(s)]⟹It,p≥0.\mathbb{E} _s\left[\max _a X _a(s)\right] \geq \max _a \mathbb{E} _s[X _a(s)] \quad\Longrightarrow\quad I _{t,p}\geq 0.

Thus, both Rt,pR_{t,p} and It,pI_{t,p} are non-negative, ensuring that Δt,p=Rt,p+It,p\Delta_{t,p}=R_{t,p}+I_{t,p} decomposes the expected local regret into two non-negative components.

Question 3. By "remaining state uncertainty", we refer to the uncertainty over the current environment state represented by the estimated belief distribution (q^t,p(s)\hat q_{t,p}(s)). Resolving this uncertainty means hypothetically obtaining access to the true state before selecting an action.

In Eq. (5), the first term represents the expected value when the true state is known, allowing the best action to be chosen separately for each state. The second term represents the value of the best single action chosen under the current belief, without knowing the true state. Their difference therefore measures the expected benefit of resolving the state uncertainty.

In particular, if the same action is optimal across all states with nonzero probability under the belief, then It,p=0I_{t,p}=0, even if the belief is highly uncertain. Conversely, when different plausible states favour different actions, knowing the true state can improve the achievable value, resulting in It,p>0I_{t,p}>0. We will clarify this interpretation in Section 2.4.

Question 4. The separate models refer to the pre-reasoning and post-reasoning value models, indexed by the token region pp in gθpg_{\theta_p}. Each model uses the activation-derived state representations from its corresponding token region and is fitted separately using the actions generated in that condition. Specifically, the pre-reasoning model uses representations and actions obtained before reasoning, while the post-reasoning model uses representations and actions obtained after reasoning. We will clarify this distinction in Section 2.3 and add an explicit reference to Appendix A.3, where the model construction and fitting procedures are described in detail.

Question 5. We agree that the maximum-entropy motivation for Eq.(1) could be stated more explicitly.

Our formulation uses a one-step maximum-entropy action-selection model, in which actions are evaluated through the inferred values of their immediate successor states rather than full trajectory returns. Specifically, defining the utility of an action as Uθ(s,a)=gθ(TG(s,a))U_\theta(s,a)=g_\theta(T_G(s,a)), we consider the entropy-regularized objective

\left[ \sum_{a\in\mathcal A} \pi(a\mid s)U_\theta(s,a) +\frac{1}{\beta}H(\pi(\cdot\mid s)) \right],$$ where $$H(\pi(\cdot\mid s)) = -\sum_{a\in\mathcal A} \pi(a\mid s)\log\pi(a\mid s),$$ and $\beta>0$ controls the trade-off between utility maximization and entropy. The solution to this objective is precisely the Gibbs (Boltzmann) policy used in Eq.(1): $$p_\theta(a\mid s,G) = \frac{ \exp\left(\beta g_\theta(T_G(s,a))\right) }{ \sum_{a'\in\mathcal A} \exp\left(\beta g_\theta(T_G(s,a'))\right) }.$$ Thus, our policy follows the maximum-entropy principle underlying Ziebart et al.(2008), specialized to immediate action choices rather than trajectory-level distributions. We will revise Section 2.3 to make this motivation and the connection to Eq. (1) explicit.
AuthorsOpenReview ↗

Additional Model Experiments (Qwen3.5-9B)

We again thank the reviewer for raising the concern regarding generalization beyond a single model. As mentioned in our previous response, we have now completed additional experiments using Qwen3.5-9B on a similar 9×99\times9 DoorKey grid layout.

We evaluated 100 trajectories, comprising 1,330 decisions. We applied the same belief–action gap framework, including the decomposition into the belief–action gap RR, the expected value of perfect state information II, and the expected local regret Δ\Delta, together with decoded belief entropy HH and the normalized gap RnormR_{\mathrm{norm}}.

The results are summarized below.

QuantityAll Steps PreAll Steps PostAll Probes Correct PreAll Probes Correct PostHeld-out PreHeld-out Post
Expected local regret Δ\Delta0.4990.5630.4900.5100.5350.762
Belief–action gap RR0.2580.1740.2750.1820.1930.143
Value of perfect state information II0.2420.3890.2150.3280.3420.619
Decoded belief entropy HH2.1722.7211.9782.3652.8914.037
Normalized gap RnormR_{\mathrm{norm}}0.3170.2820.2940.2820.3090.254

The three evaluation subsets contain 1,330 decisions (All Steps), 1,047 decisions (All Probes Correct), and 285 decisions (Held-out).

The key finding replicates across models: both RR and RnormR_{\mathrm{norm}} decrease from pre- to post-reasoning in all three subsets. This provides additional evidence that the observed reduction in the belief–action gap is not specific to GPT-OSS-20B.

An important qualitative difference between Qwen3.5-9B and GPT-OSS-20B concerns reasoning length and stability. During our Qwen experiments, we frequently observed exceptionally long reasoning traces, sometimes extending to thousands of tokens. In several cases, Qwen entered repetitive reasoning loops, appeared to lose track of its original position, and required forced termination to elicit a final action.

This behavior may help explain why Qwen exhibits a substantial increase in the expected value of perfect state information after reasoning. In the held-out subset, decoded belief entropy increases from 2.891 to 4.037, while the value of perfect state information increases from 0.342 to 0.619. We hypothesize that prolonged and repetitive reasoning may degrade the precision of Qwen's decoded state representations, distributing belief mass across states that favor different actions. This could increase the expected benefit of knowing the exact state, even as the selected action becomes more consistent with the estimated belief.

These additional results strengthen the evidence for the generality of our main belief–action gap finding across language models, while the differences in reasoning behavior highlight the importance of investigating model-specific dynamics. We will incorporate these results into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.