acceptodds
Withdrawn from ICLR 2027

Visualizing the World Model in an Agent's Value Function

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Could model-free reinforcement learning algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization agent forms a world model of the Crafter environment. Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning agents that will prevent us from building truly advanced machine intelligence.

Primary area:
interpretability and explainable AI
Keywords:
Visualization, World Models, Agents, Counterfactuals, Value Function, Crafter

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_dpDBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper uses counterfactuals on state features to explore whether value functions are implicit world models. The critic of a PPO agent is interrogated with various counterfactual observations in the Crafter environment, and the ordering of the resulting value predictions align with the authors' intuition. The authors argue this suggests the value function does form an implicit world model.

Critical strengths

The paper is well presented and easy to follow.

Critical weaknesses
  1. My main issue with this paper is that the core claim (that value functions are implicit world models) is not defined in a testable way, and the provided evidence does not support the usual meaning of 'world model' in this context. The introduction describes a world model as 'take in past inputs and predict the future which requires internalizing the dynamics of an environment' but the paper never tests whether the value function has in fact done so. The experiments simply show that the value function learns to predict expected return from an observation such that the ordering of value aligns with the authors' intuition. This is the expected behaviour of a well-fitted value function that can generalise, and doesn't support the argument that it has learned a 'world model' of the environment.
  2. Experiments are for a single trained instance of one RL algorithm (PPO), in one environment. This does not provide sufficient evidence across learning algorithms, seeds, or environments to support the general claim that the value function of model-free RL algorithms are implicit world models.
  3. Unhelpful and unnecessary anthropomorphising of the agent throughout, in my view inappropriately so. The 'depression causes suicide' paragraph just describes the expected behaviour of the agent in a state where a terminating action has the greatest expected return, so the discussion is not insightful or necessary. Similarly, section 6.1 does not add any useful insights to claimed contribution of the paper, nor is it supported by the evidence in the paper.
Questions for authors

Plant eating counterfactual (fig 6c): V(ripe plant) is greater than V(unripe plant) with high success rate from the start of the plot, while the 'plant eating' success rate remains near zero until just before the end of the graph, and only reaches 20%. Why is this? My interpretation is that the value function is 'correctly' ranking ripe vs unripe before the policy is actually eating the fruit?

Reviewer_ZCr8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This submission investigates the value function of a PPO agent trained on Crafter with the goal of understanding whether it is able to respond to changes in the state. The value function appears to reflect the ability of the agent to reach several of the achievements (e.g. minimill a diamond), and the author conclude that this means that the value function implicitly captures a world model. In particular, the author show how the value function predicts finiteness of rewards, reachability of achievements, and changes in the character's health. Each of these findings is further validated to better understand it.

Critical strengths
  • This submission is overall well written and easy to understand.
  • The experimental section is insightful, and does not stop at benchmarking.
Critical weaknesses
  • This work does not clearly state what it means for the agent to "form a world model", and it is not entirely clear why the counterfactual evaluation would answer whether the world model is formed or not. in my opinion, checking the ordering of states with respect to the value function merely measures whether the value function is accurate or not. In other words, an optimal value function (which could be computed exactly for a small discrete MDP) would also display the properties highlighted in Section 5. All the evaluation does is measuring whether PPO learns a good value function.
  • The evaluation only covers a single algorithm and a single environment, which is presented in a non-standard way (Crafter is designed as a pixel-based environment to the best of my knowledge). While I understand that this may be necessary due to compute constraints, the evaluation remains limited. The scope of the paper is then affected: the main contribution I see is displaying that PPO learns a good value function in state-based Crafter.
  • The main technical novelty, to the best of my knowledge, is the procedure described in Algorithm 1. This however remains a simple metric, which has been used to some extent in the past (e.g. the line of work on counterfactual state explanations by Olson et al.).
  • This submission heavily anthropomorphizes the PPO agent, see e.g. lines 93, 273, 284 and Section 6.1. While this makes the writing more compelling, I do not think it is scientifically accurate; it may be a better fit for a blog post. At the end of the day, in my opinion, PPO is simply optimizing discounted returns. Interpreting the resulting behavior as symptoms of the agent's state of mind may be excessive. While inspiring, the connections to literature and philosophy in Section 6.1 are unusual in an ICLR submission.
Questions for authors
  • What is the main contribution of this work? In my understanding, I see three potential options, which I will list here, but may not match your intent. First, you may want to display that value functions in general are implicitly world models. Second, you may want to introduce a novel method for interpreting value functions. Third, you may want to show that RL policies can retrieve accurate value functions. Please, let me know what is the main contribution, and how I should interpret your work.
Reviewer_c5ozOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work aims to apply counterfactual analysis to gain insights about the behavior of the value function of a PPO agent and if it learns domain knowledge similar to the approach of world models. Therefore, it uses the learning environment Crafter, a 2D minecraft clone, and changes inventory states on the fixed final policy rollout to apply the proposed counterfactual analysis.

Critical strengths

The idea of this paper is to apply counterfactual analysis to more than just the policy. The performed and reported experiments as well as the discussion are an interesting direction of research. The symbolic state allows clean, exact interventions, and the experiments use large evaluation samples with CIs.

Critical weaknesses
  1. The concept that value functions are implicit world models is sound and an informative observation. However, I think it is also a bit trivial that this is the case. The paper defines a world model as internalizing the dynamics of the environment (p.1), but all experiments only test whether the value function reacts to reward-relevant features. Learning that diamond + iron pickaxe means reward is what any return predictor must learn; this is not evidence of a dynamics model. So the value of this work, imho, lies in conducting a deep analysis of that hidden phenomenon. However, I do not see that some eye-opening visualization technique, as promoted in the title, is ever presented.

  2. The analysis is driven too much by human expectations, e.g.: "In the diamond example, if the critic had only learned correlations, the presence of the diamond would increase the value prediction because it is associated with a positive reward. However, if the value function learned deeper rules about the environment, then the mere addition of a diamond would not significantly increase value unless it is actually exploitable, i.e., the diamond is present and the agent has the iron pickaxe." (p.3.) seems superficial to me. The idea of using counterfactual analysis to identify cause vs. correlation is great. But any function approximator trained on returns will learn conjunctions like diamond + iron pickaxe, so this example on human-like-thinking-based semantics does not distinguish correlation from "deeper rules". I do not think the diamond+pickaxe rule can be called "deeper" understanding. The introduction started with similar superficial statements. The climax of it is the philosophical discussion on Viktor Frankl's "Man's Search for Meaning", which is interesting, but imho not suitable for ICLR in this form.

  3. The experiment scenarios are very limited. First, it just plots one episode (Fig.2) and calls it a value function collapse. The inventory-swap test generalizes this, but the decline itself is expected: with finite reward and γ=0.99\gamma = 0.99, a correct value function should decline as achievements are used up. Then the paper asks "Why is this?" and answers with more questions, which are never discussed further. Same limitations are true for using only a few hand-designed counterfactual test cases (inventory, diamond/iron/plant, water/food). Many of these counterfactuals (e.g., removing the iron pickaxe while facing a diamond) create states the policy rarely visits, so the critic's predictions there are weak evidence. Further, the screenshots are very hard to understand for a reader who is not familiar with that game.

  4. There is no related work section.

Questions for authors
  1. I am missing tests on statistical robustness. Fig.2 & Fig.6 do not report std or CI, and the CIs in App.C are over states/episodes of a single trained agent, not over training runs. App.B lists only one seed (=50). And I found "Using THE final policy checkpoint" on p.3. Please clarify if there really was only one training run.

  2. What was your reasoning on choosing these hyperparameters presented in App.B? In particular, do the findings (value decline, "depressed states") still hold for other values of γ\gamma, since they hinge on γ=0.99\gamma = 0.99?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.