- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors aim at understanding how much can memory based RL driving policies trained solely on synthetic data generalize to human policies. They thus train such policies on synthetic data and evaluate how much these adapts to human policies. They show that these memory based policies are bad at such tasks when memory is deactivated. Then they use attention map to try to interpret how agents adapt.
Critical strengths
The paper is overall clear in the main problem it is addressing: Can we train RL based driving policy only on synthetic data and easily generalize (in few shot) to human data?
I did not need to ask an LLM to understand what the paper does and how globally (which is rare nowadays), even if some details are not clearly exposed.
Critical weaknesses
I have trouble extracted the main takeaway message(s). What is the core takeaway message from this paper that I can apply to other AI domains? Or that give me a new perspective on this specific problem?
My main concern is that I don't feel like the hours spent on the paper brought me any clear takeaway message that I can reuse in discussions with other researchers or within my own research.
First, I was not convinced that ablating out of the training set human data is a necessity. Second, there is nothing compared to. Other baselines would help understand the advantage of one method over another one. You ablate memory from a memory-based RL policy to then deduce that it is a crucial part of it (am I really learning something here).
I am not saying that the research pursued has no value, simply that I was not able to extract it.
I am extremely doubtful about the conclusions on using attention maps to extract interpretations. This is not a core cause of my judgement of the overall paper, but I would like to bring the authors to think about the following: Attention maps do not allow you to draw conclusions on an interpretation of what is happening. It tells you that a specific part of the input is important in the decision process, but not why is this input part important. In the second reference below, importance maps show the ball as a relevant part of the input, however, the agent is clearly using the enemy's position (spuriously correlated) to infer the ball's position (if it does infer the ball position). Interpreting the reasoning behind the heatmap is still an ad-hoc interpretation that can be proven to be wrong (causality checks, at least, would be needed here).
Di Langosco, L. L., Koch, J., Sharkey, L. D., Pfau, J., & Krueger, D. (2022, June). Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning (pp. 12004-12019). PMLR.
Delfosse, Q., Sztwiertnia, S., Rothermel, M., Stammer, W., & Kersting, K. (2024). Interpretable concept bottlenecks to align reinforcement learning agents. Advances in Neural Information Processing Systems, 37, 66826-66855.
Clarity is not perfect though: "We ask whether such adaptation can be learned without any human data." -> but you use memory over replayed episodes of human data, so somehow it's learned with human data.
The scope is quite narrow though (only autonomous driving), there is no study of the distance between the synthetic policies and the human ones (which would allow me to bring a broader conclusion). This is a bit redundant with the takeaway message(s) not being easy to identify.
Questions for authors
Not saying non of this is present in the full text of the paper. But after reading it fully, I was confused and tried to use only the abstracts, contributions and figures to answer these, but could not:
- In Figure 1 left, why are there so many ego agents? (even if caption only talks about a single one).
- Policies are getting ego-centric states and goal, but how are they encoded? A frozen causal transformer is use? Not trained on your data? But pretrained? So only natural language policy is used? (that should go in your method, Figure2).
- Co plauyers are pre-scripted policies, or are they independant copies of the ego?
- Are you training a memory based system? And then demonstrate the importance of memory?