- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to disentangle an LLM agent's belief, preference and action.
The authors tackle this problem with inverse RL (to estimate the value functions) and linear probes (to decode model's internal belief of the world).
I would say that the main contribution of this paper is the creative use of the above tools to understand the "belief-action gap" in an LLM agent. The experiment design to probe representations before and after reasoning trace is also thoughtful and reveals that reasoning help closes the belief-action gap.
Experiments are done in a grid world environment, so it lacks diversity and realism, but it is a good controlled environment to study the problem formulated here. I don't find it a major concern.
I do have some concerns regarding the validity of the approach. I will detail them in later sections.
Critical strengths
The experiments are well-designed. First block of experiment measure the functions, whereas second block understands CoT, and the last block explores intervention and seeks for causal proof of previous findings. Interesting empirical findings are shown, demonstrating that <reasoning> helps <aligning action with belief and value>, but not <forming a more accurate belief>.
I also like that the paper studies the effect of CoT reasoning (introducing a 4th variable other than belief, value, and action), by injecting probing questions about next action in the middle of the CoT traces. The result that model forms their final action preference in the middle of the reasoning trace is surprising (though aligned with other relevant work in LLM response planning).
Critical weaknesses
One concern is the validity of the approach.
The Inverse RL model uses the state representations and the observed action to infer preferences, but the model may actually be acting on an inaccurate belief about the state! You are not decoding the belief from the model in this step, so you won't know the model's belief when fitting the preference model. If so, an action mistake caused by an incorrect belief could be interpreted as a preference error. This makes it unclear how well the framework can actually disentangle these two sources of failure.
Another concern is the lack of reference for closely related work. I don't know if there's more, but here are two missing papers that are very relevant: [1] also decodes beliefs from the representations of LLMs and intervene on them, and [2] (which is from ICLR 2026) discusses a more realistic belief-action gap where the model can infer the belief but fail to infer the action led by the belief.
Moreover, a minor weakness is the presentation. I find a lot of terms in the paper not well-defined, especially not explained in their first appearances. E.g., I have to read more and do in-context learning to understand "state preference" is the value function (real preference), and "action preference" is the potential action distribution of the agent. "Pre-reasoning" and "post-reasoning" also appears early in 2.3 but not explained until later sections
[1] Zhu, W., Zhang, Z., & Wang, Y. (2024). Language models represent beliefs of self and others. arXiv preprint arXiv:2402.18496.
[2] Gu, Y., Tafjord, O., Kim, H., Moore, J., Le Bras, R., Clark, P., & Choi, Y. (2026, April). Simpletom: Exposing the gap between explicit tom inference and implicit tom application in llms. In International Conference on Learning Representations (Vol. 2026, pp. 50252-50297).
Questions for authors
In the wall-hit example, the model has the correct belief about the world but chooses the incorrect action. You suggest that this is due to the belief-action gap. I don't quite get it because if the model looks forward for one single step in their reasoning trace, they would know that there is a wall. So I was wondering if this could be due to the model's incorrect belief of the world transition function (which is not measured in this work)?
3 comments
Thanks for the response! The rebuttal addresses most of my concerns. I like the design of the experiments and the methodology, though more limitations need to be acknowledged, e.g., the method cannot reliably distinguish between belief error or preference error. It would be desirable if you could derive a more accurate method in disentangling different error sources. And yes please discuss these relevant works.
I have increased my rating because I think this work is novel and interesting, but my concern about the approach remains, so I lowered my confidence at the same time. It really depends on different people's internal value functions to position this paper.
We thank the reviewer for their thoughtful follow-up and for reconsidering our work. We appreciate the positive assessment of our experimental design and methodology.
We appreciate the reviewer's constructive feedback, which has helped us clarify the scope and limitations of our approach.
We thank the reviewer for these constructive comments. We address the methodological concern, the suggested references, and the terminology below.
Belief--preference confounding in IRL. We agree that an agent may select an action based on an incorrect belief about the environment state. Since our one-step IRL model is fitted using the observed environment states and the agent's actions, the inferred state values may partially reflect the consequences of inaccurate beliefs rather than preferences alone. This is an important limitation of inferring preferences from observed behaviour.
Our framework does not assume that the inferred state values uniquely recover the agent's internal preferences. Rather, the IRL model provides an operational estimate of the state preferences expressed through the agent's behaviour. We then combine these inferred values with independently estimated belief distributions to quantify the consistency between represented state information and selected actions.
In particular, our decomposition distinguishes the expected loss associated with the emitted action under the estimated belief from the expected value of resolving state uncertainty. These quantities are well-defined under the fitted belief and value models, but we do not claim that they uniquely identify the underlying causes of individual action errors.
Importantly, our results are not restricted to the observations used to fit the estimators. As reported in Table 1, the belief--action gap decreases from 0.316 to 0.199 on the jointly held-out subset of 762 decisions, whose observations were excluded from both probe and IRL training. This supports the generalization of the measured relationship, although it does not by itself eliminate belief--preference confounding.
An alternative would be to fit the IRL model directly using the estimated belief distributions rather than the true environment states. However, this would make preference inference dependent on the accuracy and calibration of the belief probes, potentially introducing additional estimation errors.
Relationship to prior work. We thank the reviewer for pointing out these relevant references, which we will include into the revised manuscript.
Zhu et al. (2024) investigate representations of agents' beliefs in language models and demonstrate that belief-related information can be decoded from internal activations and influenced through activation interventions. This work is related to the representational and causal aspects of our study. Our focus, however, is on the relationship between an agent's estimated environment-state beliefs, its inferred state preferences, and its own action selection. In particular, we introduce a quantitative decision-theoretic diagnostic that combines activation-derived beliefs with IRL-inferred values.
Gu et al. (2026) demonstrate a gap between explicit Theory-of-Mind inference and the application of that information in predicting or evaluating behaviour. This finding is conceptually related to our motivating question of whether represented information is consistently reflected in decisions. Our setting differs in that we study an agent's own environment-state beliefs and actions rather than its predictions about another person's behaviour. We additionally quantify action consistency through expected value differences under estimated beliefs and inferred preferences, and investigate how this relationship evolves during reasoning.
Clarity of terminology. We agree that several terms should be defined more clearly when first introduced. We will revise the presentation to distinguish state preferences, referring to the scalar state values inferred through IRL, from action preferences, referring to the model's elicited probability distribution over candidate actions. We will also define pre-reasoning and post-reasoning at their first appearance, clarifying that they refer to measurements obtained before and after the model generates its reasoning, respectively.
Question 1. We thank the reviewer for this observation. We agree that an agent may correctly identify its current environment state while incorrectly predicting the consequences of an action. Such an error could contribute to the observed wall-hit failures. Our analysis in Appendix B.1 examines three wall-hit decisions in which the model selects a direction that is blocked by a wall. In all three cases, the model correctly reports that the selected direction is blocked. These examples demonstrate that the local state information needed to avoid the error is explicitly recoverable, even when the agent ultimately selects the blocked action. Importantly, we do not interpret these results as establishing that the model necessarily used this information during its original decision. As discussed in Appendix A.2, report-based beliefs are elicited through separate forward passes and may recover information that was not used when selecting the action.