- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows that agents' behavior at deployment needs to be conditioned on the history of interaction, in order not to cause the number of parameters that the agent needs to become impractical. The argument is roughly that if the policies can't adapt, then they need to know the whole world model before deployment. The paper shows that in order to figure out the world model describing a given environement a fixed policy requires a lot of parameters. Then they show that in their theoretical framework, it is possible to trade-off the parameters for ingesting experience at inference.
Critical strengths
I believe that in reinforcement learning community many people still train agents that are not adaptable. This paper provides a strong signal for the community to move into the direction where adaptability is more important than just offline performance.
Critical weaknesses
I believe that the actual finding of the paper is interesting. However the introduction diverges from the true story, through using language models as an example of models that aren’t adaptive. Which they clearly are. For instance, according to definition 1, LLMs are not frozen as they are autoregressive. That makes Figure 1 confusing and the whole introduction to be as though it's from a different paper than the rest.
There is no real experimental section in this paper. I know that training RL agents in the offline to online regime tends to not be that easy, but an experimental result of this type, or alternatively one comparing an agent with and without history would make the paper much more complete.
Questions for authors
- In practice, how do we adapt online without knowing the environment before the adaptation phase?
- Could you spell out the connection to language models implied in the introduction?
- How would you validate your theory experimentally?
- Could the authors present even a very toy experiment supporting the theoretical findings?
- If the real world is governed by patterns, that is, I expect that in each house, the fridge is somewhere close to the stove, do the theoretical results describe the real world accurately?