- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper uses counterfactuals on state features to explore whether value functions are implicit world models. The critic of a PPO agent is interrogated with various counterfactual observations in the Crafter environment, and the ordering of the resulting value predictions align with the authors' intuition. The authors argue this suggests the value function does form an implicit world model.
Critical strengths
The paper is well presented and easy to follow.
Critical weaknesses
- My main issue with this paper is that the core claim (that value functions are implicit world models) is not defined in a testable way, and the provided evidence does not support the usual meaning of 'world model' in this context. The introduction describes a world model as 'take in past inputs and predict the future which requires internalizing the dynamics of an environment' but the paper never tests whether the value function has in fact done so. The experiments simply show that the value function learns to predict expected return from an observation such that the ordering of value aligns with the authors' intuition. This is the expected behaviour of a well-fitted value function that can generalise, and doesn't support the argument that it has learned a 'world model' of the environment.
- Experiments are for a single trained instance of one RL algorithm (PPO), in one environment. This does not provide sufficient evidence across learning algorithms, seeds, or environments to support the general claim that the value function of model-free RL algorithms are implicit world models.
- Unhelpful and unnecessary anthropomorphising of the agent throughout, in my view inappropriately so. The 'depression causes suicide' paragraph just describes the expected behaviour of the agent in a state where a terminating action has the greatest expected return, so the discussion is not insightful or necessary. Similarly, section 6.1 does not add any useful insights to claimed contribution of the paper, nor is it supported by the evidence in the paper.
Questions for authors
Plant eating counterfactual (fig 6c): V(ripe plant) is greater than V(unripe plant) with high success rate from the start of the plot, while the 'plant eating' success rate remains near zero until just before the end of the graph, and only reaches 20%. Why is this? My interpretation is that the value function is 'correctly' ranking ripe vs unripe before the policy is actually eating the fruit?