- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors investigate why RL agents are subject to shortcut learning (or misalignment in RL). They show that identical shortcut learning behavior can be caused by 3 different mechanisms!
- functional Target Substitution: the goal is absent from the network's internal representation
- policy shortcut: encoded in visual features but discarded before being given to the policy,
- cue competition: the target is provided to the policy but discarded because of another visual cue.
They show that the success of post-hoc alignment interventions depend on the depth at which the information is lost: activation steering at the final layer resolves cue competition, training a new readout head over frozen features repair "policy shortcut" and neither approach can fix target substitution..
Critical strengths
The addressed problem is crucial: misalignment is of great importance. It is a greatly underestimated problem by RL practitioners. Unfortunately, we, as the RL community still train and test in mostly the same environment, varying the action noise, and expect reasoning policies (i.e. trained on reasoning tasks such as Atari and Procgen) to learn to select the right action for the right reason. They do not.
Critical weaknesses
Clarity [addressed]
I find that the clarity, specifically to a broader audience, can be greatly improved. I -as an expert of goal misalignment in RL- already struggled to understand the main points. To help the authors improve clarity. Here is how I read a paper. Before fully reading the main text, I read the abstract, then go to Figure 1, contributions, all the other figures and conclusion. In my opinion, this should allow me to get 95% of your paper. The rest is if I want to reimplement it, understand specific details to make sure the experimental evaluation is correct, ...etc.
Give your paper to an AI expert (no misalignment, no RL deep knowledge), and ask them after such a reading, what did they understood from your paper. I'll bet that they'll have not understood more than 50%, and this is a shame, as the questions you're raising and answering is extremely important, particularly for ICLR readers.
I also advice the authors to have a first figure that only depicts the addressed problem. This problem is complex enough, so only have this first figure detail the difference between functional target substitution, policy shortcut, and cue competition. Also, here you have one example of each. Can you use other Procgen environments or e.g. JAXAtari to create other examples of this. I think that it would make it easier to grasp the difference between the 3 if they appeared onto the same environment (probably easily to do with JAXAtari+a coding agent).
Distinction is not clear to me. [discussed, I got convinced that the paper's argumentation is beneficial]
The distinction between the 3 types of misalignment is interesting, however, I don't understand the clear distinction between (1) and (2). In general, each type of misalignment is due to correlation between cues. In general you have these types of cues, that can be correlated:
- global position of the target (coin at the end in coinrun or top right in your maze)
- speed
- color
- shape
- orientation
- ... more generally, the look of the object (in our case, the sprite) If two of these cues are correlated, the neural network risks to base its decision on either the wrong one, or on a correlation between the correct and potential incorrect causal factors.
Where does the representation of each cue emmerge in your architecture is very interesting in order to correct this misalignment, but I think that these 3 types should be rethought using this correlation between different cues frame.
After reading the paper and my review a second time, I think that the problem is that the naming does not make obvious that what you are interested is the layer at which the critical causal cue is discarded. Maybe you should rename the option to better express this.
Finally, there are other strategies that are not really tested or discussed here (or only briefly in the discussion):
- What about using humans (or LLMs) to manually discard irrelevant competing cues?
- What about reward shaping? (although I don't find it that meaningful here, but can be used to mitigate misalignment)
- What about inductive bias (e.g. LLM generated python policies on symbolic states)?
Questions for authors
- Can you add the correctly aligned policy results as a baselines? (even if it is always 1)
- What is GM, non-GM in Figure 2 ? (I know, this is provided in the main text, but a figure should be standalone).
- Why not using JAXAtari? It will also allow you to have extremely nice interventions on shortcuts learning environments (pong with the enemy, freeway with the car colors, ...etc). (Note that this is not a strict necessity for your paper to be demonstrating your points, but using a second benchmark will greatly help harden the quality of your evaluation).
3 comments
Thank you. Small feedback still
Thank you for your answer. First, thank you for all the additional work. I still believe that JAXAtari would have allowed you better exemplification than the robotics experiments, but adding a domain is not hurting, and we cannot request all possible experiments. I believe that the current set of experimental evaluations is sufficient to support the authors claims.
I looked at the updated manuscript, it has greatly been improved but is not fully yet to a stage where I fully feel comfortable raising (but we're getting there). I initially agreed with reviewer Vv7U on the clarity problem, but in my stack of ICLR reviewed papers:
- Most of the other papers are even worst in terms of clarity
- At least on this paper, I clearly learned something interesting, and expect researchers with less expertise in GM to gain substancial understanding (if clarity becomes a non-issue).
Let me help finetuning the last version:
For Figure 1 (nit, but easy). Make the examples bigger, use more flashy colors for the arrows (light green and red, that also align with the intuition behind the correct goal and shortcut). The Figure is too small and difficult to read without zooming a lot.
Figure 2 would benefit from a more detailed caption, that would insist on the first takeaway message more. What are the 3 lines showing exactly? Again, take your paper from the perspective of the reader who would only read the Figures, not the details in the text, it would be hard for them to see what is the claim/takeaway message of Figure 2 (after only reading figure 1).
Also, what is the main takeaway message of the paper in general? That shortcut learning/GM can be due to different causes, and that each cause require a different intervention? Maybe extracting this one takeaway and placing it with bold font in the contribution would help the reader extract what to assess. Then the first bold sentences of the figure should be aligned with this main claim. Take a step back, try to really set yourself in the shoes of a non expert of your paper, and go through 1. the main claim/takeaway messages and all the Figures + tables and make sure they are aligned, and that the reader can understand them without having to ask an LLM or to look into the details of the text.
I know that I am asking changes again, but I am really inclined towards raising, but I would like the paper to be easily understandable to many RL practitionners that underestimate the impact of GM.
Thank you very much for your fast reply, and we are always very happy to hear what can be improved further. We applied your suggestions right away and uploaded a new revision.
Figure 1. The figure now uses stronger colors, green for the goal and red for the proxy, and is simplified so that the examples are much bigger and the figure is easy to read without zooming. We removed the experiment numbers from Figure 1, since they are shown in Figures 5 and 6, so that the figure only conveys the problem and the main finding: which recovery works depends on where the goal is lost. Here the improved caption:
Figure 1: Which recovery works depends on where the goal is lost. Left: in goal misgeneralization (GM), the goal (green, solid ring) and the proxy (red, dashed ring) coincide in training. At test they separate, and GM agents (burgundy) follow the proxy, for example the top-right corner instead of the cheese in Maze, while generalized (Non-GM) agents (teal) follow the goal. Right: we find that the goal is lost at one of three depths: the policy does not read it, reads it but does not follow it, or follows it while the proxy outcompetes it. Steering the existing policy recovers only agents that follow the goal, and rewiring, which relearns the last layers, only agents whose policy reads it. Results in Figures 5 and 6, all eight environments in Figure 13.
Figure 2. We rewrote the caption so that it states the takeaway first and explains what the lines show:
Figure 2: The goal is decodable in GM agents of Coin and Attribute Maze, but not in Maze and PushToMarker, so goal information can remain in GM agents. Lines: how well a linear probe decodes the goal per layer in GM (burgundy), Non-GM (teal) and untrained networks of the same architecture (grey). A GM line at the grey line holds no more goal information than an untrained network.
Figure 4. Following your advice to read the figures from the perspective of a non-expert, we also reordered and simplified Figure 4, so that it can be understood with as little additional explanation as possible:
Figure 4: In Attribute Maze, GM agents fail through cue competition, a push and pull (a) along a shared axis (b), which the proxy wins (c), so the goal is followed, but outcompeted. (a) A goal swap makes the goal object the yellow line, a proxy swap the proxy object. The object that receives the yellow line gains about as much log preference as the other loses, a push and pull. (b) Fisher-weighted cosine between states of their competition direction: the representation shift of the proxy swap minus that of the goal swap. Near , states share one axis, its sign set by whether the goal is up or right. (c) Pull , each swap's shift along this shared axis .
Main finding. Lastly, we added a bold main finding right above the contributions, to show exactly what we want the reader to take away, and aligned it with the message of Figure 1:
Therefore, our main finding is that goal-misgeneralized agents can fail in the same way, but still need different recovery methods, and which recovery method works depends on the depth at which the goal is lost.
We also want to note that we are really excited about your engagement and your help in fine-tuning the paper, which we truly appreciate. It is great to see that you want to make it as good as it can be. So asking for changes again is not a problem at all, and we are very happy to make any further edits or changes that might still be needed. Thank you again for taking the time to help us make the paper easier to understand for RL practitioners, this is exactly who we want to reach.
We thank you for this detailed review, and especially for the concrete advice on making the paper more understandable. We rewrote the abstract, introduction, all figures and captions, the conclusion and much of the main text, so that the skimmable parts stand on their own. Each figure now has a caption, readable legends, titles and labels that make it self-explanatory, with direct callbacks to the abstract, the main findings and the contributions. We also sharpened the abstract and the conclusion to align them with the stronger findings, which also improves clarity.
We would be grateful for feedback on the revised version, in case anything is still unclear.
First figure should depict the problem → Added the problem to Figure 1, and kept the main finding next to it. Figure 1 now shows the problem with concrete examples (panel a) and the main finding (panel b): the goal is lost at different depths, and the depth decides how it can be recovered. It deliberately leaves out the three failure types. The depth became the main finding because it predicts which recovery works, across environments, which a classification by failure type does not.
More examples, other environments → Added robotics: five new environments, eight in total. Instead of more game environments, we went beyond Procgen and added four robot-arm tasks (ManiSkill3) and a quadruped task (Isaac Lab):
Distinction between (1) and (2), cue-correlation frame → The depth at which the critical cue is lost is now the main axis. The failure types remain a second axis. We agree! The focus is now on the depth as a classifier of GM environments. Noting here that the depth describes how the policy uses the goal, not a network layer.
Other strategies → Interesting, but outside the scope of this paper.
Q1. Correctly aligned policy as a baseline → Yes, the Non-GM agents are used as reference in every result. In the main section, every figure now includes them and table 1 notes in the caption now that it is a ratio with respect to the Non-GM agents and therefore the Non-GMs have value 1 here.
Q2. GM and Non-GM in Figure 2 → The legend now shows GM (misgeneralized) and Non-GM (generalized).
Q3. Why not JAXAtari? → We prioritized robotics as a second domain. Thank you again for the suggestion! Instead of a second game benchmark we extended the test of generality even to a second domain.