acceptodds
Under review as a conference paper at ICLR 2027

What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization

Abstract

Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. From the outside these failures look alike, yet we still understand little about what happens to the intended goal inside misgeneralized agents. We trace the goal through agents in three reinforcement-learning environments, asking whether it is still represented, whether the policy reads it out, and what organizes behavior instead. We find three distinct failures of policy alignment, in which the goal gets lost at different depths: (1) functional target substitution, where the goal is inert to the policy and behavior follows an alternative target, (2) a policy shortcut, where the goal is represented but not used as a goal and behavior follows a learned action rule, and (3) cue competition, where the goal reaches the decision but is outweighed by a proxy cue. Where the goal gets lost explains which recovery works. Under cue competition, steering along the axis on which the two cues compete raises intended-goal reach from to , while the shortcut recovers only after learning a new readout, from to , and under target substitution neither restores goal-directed behavior. So goal misgeneralization is not one failure, and recovering the goal requires knowing where it got lost.

Primary area:
reinforcement learning
Keywords:
goal misgeneralization, reinforcement learning, mechanistic interpretability, representation analysis, goal representations, AI alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ec12OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The authors investigate why RL agents are subject to shortcut learning (or misalignment in RL). They show that identical shortcut learning behavior can be caused by 3 different mechanisms!

  1. functional Target Substitution: the goal is absent from the network's internal representation
  2. policy shortcut: encoded in visual features but discarded before being given to the policy,
  3. cue competition: the target is provided to the policy but discarded because of another visual cue.

They show that the success of post-hoc alignment interventions depend on the depth at which the information is lost: activation steering at the final layer resolves cue competition, training a new readout head over frozen features repair "policy shortcut" and neither approach can fix target substitution..

Critical strengths

The addressed problem is crucial: misalignment is of great importance. It is a greatly underestimated problem by RL practitioners. Unfortunately, we, as the RL community still train and test in mostly the same environment, varying the action noise, and expect reasoning policies (i.e. trained on reasoning tasks such as Atari and Procgen) to learn to select the right action for the right reason. They do not.

Critical weaknesses

Clarity [addressed]

I find that the clarity, specifically to a broader audience, can be greatly improved. I -as an expert of goal misalignment in RL- already struggled to understand the main points. To help the authors improve clarity. Here is how I read a paper. Before fully reading the main text, I read the abstract, then go to Figure 1, contributions, all the other figures and conclusion. In my opinion, this should allow me to get 95% of your paper. The rest is if I want to reimplement it, understand specific details to make sure the experimental evaluation is correct, ...etc.

Give your paper to an AI expert (no misalignment, no RL deep knowledge), and ask them after such a reading, what did they understood from your paper. I'll bet that they'll have not understood more than 50%, and this is a shame, as the questions you're raising and answering is extremely important, particularly for ICLR readers.

I also advice the authors to have a first figure that only depicts the addressed problem. This problem is complex enough, so only have this first figure detail the difference between functional target substitution, policy shortcut, and cue competition. Also, here you have one example of each. Can you use other Procgen environments or e.g. JAXAtari to create other examples of this. I think that it would make it easier to grasp the difference between the 3 if they appeared onto the same environment (probably easily to do with JAXAtari+a coding agent).

Distinction is not clear to me. [discussed, I got convinced that the paper's argumentation is beneficial]

The distinction between the 3 types of misalignment is interesting, however, I don't understand the clear distinction between (1) and (2). In general, each type of misalignment is due to correlation between cues. In general you have these types of cues, that can be correlated:

  • global position of the target (coin at the end in coinrun or top right in your maze)
  • speed
  • color
  • shape
  • orientation
  • ... more generally, the look of the object (in our case, the sprite) If two of these cues are correlated, the neural network risks to base its decision on either the wrong one, or on a correlation between the correct and potential incorrect causal factors.

Where does the representation of each cue emmerge in your architecture is very interesting in order to correct this misalignment, but I think that these 3 types should be rethought using this correlation between different cues frame.

After reading the paper and my review a second time, I think that the problem is that the naming does not make obvious that what you are interested is the layer at which the critical causal cue is discarded. Maybe you should rename the option to better express this.


Finally, there are other strategies that are not really tested or discussed here (or only briefly in the discussion):

  1. What about using humans (or LLMs) to manually discard irrelevant competing cues?
  2. What about reward shaping? (although I don't find it that meaningful here, but can be used to mitigate misalignment)
  3. What about inductive bias (e.g. LLM generated python policies on symbolic states)?
Questions for authors
  1. Can you add the correctly aligned policy results as a baselines? (even if it is always 1)
  2. What is GM, non-GM in Figure 2 ? (I know, this is provided in the main text, but a figure should be standalone).
  3. Why not using JAXAtari? It will also allow you to have extremely nice interventions on shortcuts learning environments (pong with the enemy, freeway with the car colors, ...etc). (Note that this is not a strict necessity for your paper to be demonstrating your points, but using a second benchmark will greatly help harden the quality of your evaluation).
3 comments

We thank you for this detailed review, and especially for the concrete advice on making the paper more understandable. We rewrote the abstract, introduction, all figures and captions, the conclusion and much of the main text, so that the skimmable parts stand on their own. Each figure now has a caption, readable legends, titles and labels that make it self-explanatory, with direct callbacks to the abstract, the main findings and the contributions. We also sharpened the abstract and the conclusion to align them with the stronger findings, which also improves clarity.

We would be grateful for feedback on the revised version, in case anything is still unclear.

First figure should depict the problem → Added the problem to Figure 1, and kept the main finding next to it. Figure 1 now shows the problem with concrete examples (panel a) and the main finding (panel b): the goal is lost at different depths, and the depth decides how it can be recovered. It deliberately leaves out the three failure types. The depth became the main finding because it predicts which recovery works, across environments, which a classification by failure type does not.

More examples, other environments → Added robotics: five new environments, eight in total. Instead of more game environments, we went beyond Procgen and added four robot-arm tasks (ManiSkill3) and a quadruped task (Isaac Lab):

Target substitutionPolicy shortcutCue competition
Not readMaze, LocationReach
Read, but not followedPushToMarkerCoin
Followed, but outcompetedAttribute Maze, AttributeTable, SortZones, WalkToFlag
  • Why robotics? Robotics is entering its RL phase, will face this problem as well, and goal misgeneralization has been understudied there. We also induced a policy shortcut on the robot arm, partly (Appendix F.3).

Distinction between (1) and (2), cue-correlation frame → The depth at which the critical cue is lost is now the main axis. The failure types remain a second axis. We agree! The focus is now on the depth as a classifier of GM environments. Noting here that the depth describes how the policy uses the goal, not a network layer.

  • To separate (1) and (2): a target substitution orients the behavior toward a new target, e.g., a place. For example, a maze GM-agent takes detours to reach the top-right corner, using all directions (Appendix D.4.1). However, a policy shortcut does not orient the agent toward a new target, but instead sharpens the action distribution toward a subset of actions. E.g., the coin GM-agents are biased toward going right, so they would even run past the end of the level.
  • On the cue-correlation frame, if we understood it correctly, our discussion might point in the same direction: goal and proxy cues on one object, or on two objects close together (new find in our quadruped task) seem to lead to competition deep in the agent, while location-based proxies lose the goal early. We deliberately did not investigate a taxonomy on cue types, because showing that a cue type causes a depth would require causal ablations of every cue type, and the space of cues is very large. The depth frame was a deeper finding and even predicts recovery.

Other strategies → Interesting, but outside the scope of this paper.

  • The strategies mentioned change the input or the policy class before or during training, and assume the competing cues are known and available as symbolic states. Our agents learn end to end from pixels, and in goal misgeneralization the proxy is not necessarily known in advance. We mainly use steering and rewiring as diagnostics, not as deployment-ready mitigations (see Discussion).
  • Reward shaping acts only during training, where goal and proxy coincide in every state, so any shaped reward for the goal also rewards the proxy. Separating them would require knowing the proxy and states in which it differs from the goal, which amounts to changing the training environment (Appendix I).

Q1. Correctly aligned policy as a baseline → Yes, the Non-GM agents are used as reference in every result. In the main section, every figure now includes them and table 1 notes in the caption now that it is a ratio with respect to the Non-GM agents and therefore the Non-GMs have value 1 here.

Q2. GM and Non-GM in Figure 2 → The legend now shows GM (misgeneralized) and Non-GM (generalized).

Q3. Why not JAXAtari? → We prioritized robotics as a second domain. Thank you again for the suggestion! Instead of a second game benchmark we extended the test of generality even to a second domain.

Reviewer_Ec12OpenReview ↗

Thank you. Small feedback still

Thank you for your answer. First, thank you for all the additional work. I still believe that JAXAtari would have allowed you better exemplification than the robotics experiments, but adding a domain is not hurting, and we cannot request all possible experiments. I believe that the current set of experimental evaluations is sufficient to support the authors claims.

I looked at the updated manuscript, it has greatly been improved but is not fully yet to a stage where I fully feel comfortable raising (but we're getting there). I initially agreed with reviewer Vv7U on the clarity problem, but in my stack of ICLR reviewed papers:

  1. Most of the other papers are even worst in terms of clarity
  2. At least on this paper, I clearly learned something interesting, and expect researchers with less expertise in GM to gain substancial understanding (if clarity becomes a non-issue).

Let me help finetuning the last version:

For Figure 1 (nit, but easy). Make the examples bigger, use more flashy colors for the arrows (light green and red, that also align with the intuition behind the correct goal and shortcut). The Figure is too small and difficult to read without zooming a lot.

Figure 2 would benefit from a more detailed caption, that would insist on the first takeaway message more. What are the 3 lines showing exactly? Again, take your paper from the perspective of the reader who would only read the Figures, not the details in the text, it would be hard for them to see what is the claim/takeaway message of Figure 2 (after only reading figure 1).

Also, what is the main takeaway message of the paper in general? That shortcut learning/GM can be due to different causes, and that each cause require a different intervention? Maybe extracting this one takeaway and placing it with bold font in the contribution would help the reader extract what to assess. Then the first bold sentences of the figure should be aligned with this main claim. Take a step back, try to really set yourself in the shoes of a non expert of your paper, and go through 1. the main claim/takeaway messages and all the Figures + tables and make sure they are aligned, and that the reader can understand them without having to ask an LLM or to look into the details of the text.

I know that I am asking changes again, but I am really inclined towards raising, but I would like the paper to be easily understandable to many RL practitionners that underestimate the impact of GM.

Thank you very much for your fast reply, and we are always very happy to hear what can be improved further. We applied your suggestions right away and uploaded a new revision.

Figure 1. The figure now uses stronger colors, green for the goal and red for the proxy, and is simplified so that the examples are much bigger and the figure is easy to read without zooming. We removed the experiment numbers from Figure 1, since they are shown in Figures 5 and 6, so that the figure only conveys the problem and the main finding: which recovery works depends on where the goal is lost. Here the improved caption:

Figure 1: Which recovery works depends on where the goal is lost. Left: in goal misgeneralization (GM), the goal (green, solid ring) and the proxy (red, dashed ring) coincide in training. At test they separate, and GM agents (burgundy) follow the proxy, for example the top-right corner instead of the cheese in Maze, while generalized (Non-GM) agents (teal) follow the goal. Right: we find that the goal is lost at one of three depths: the policy does not read it, reads it but does not follow it, or follows it while the proxy outcompetes it. Steering the existing policy recovers only agents that follow the goal, and rewiring, which relearns the last layers, only agents whose policy reads it. Results in Figures 5 and 6, all eight environments in Figure 13.

Figure 2. We rewrote the caption so that it states the takeaway first and explains what the lines show:

Figure 2: The goal is decodable in GM agents of Coin and Attribute Maze, but not in Maze and PushToMarker, so goal information can remain in GM agents. Lines: how well a linear probe decodes the goal per layer in GM (burgundy), Non-GM (teal) and untrained networks of the same architecture (grey). A GM line at the grey line holds no more goal information than an untrained network.

Figure 4. Following your advice to read the figures from the perspective of a non-expert, we also reordered and simplified Figure 4, so that it can be understood with as little additional explanation as possible:

Figure 4: In Attribute Maze, GM agents fail through cue competition, a push and pull (a) along a shared axis (b), which the proxy wins (c), so the goal is followed, but outcompeted. (a) A goal swap makes the goal object the yellow line, a proxy swap the proxy object. The object that receives the yellow line gains about as much log preference as the other loses, a push and pull. (b) Fisher-weighted cosine between states of their competition direction: the representation shift of the proxy swap minus that of the goal swap. Near ±1\pm1, states share one axis, its sign set by whether the goal is up or right. (c) Pull αF\alpha_F, each swap's shift along this shared axis ucompu_{\mathrm{comp}}.

Main finding. Lastly, we added a bold main finding right above the contributions, to show exactly what we want the reader to take away, and aligned it with the message of Figure 1:

Therefore, our main finding is that goal-misgeneralized agents can fail in the same way, but still need different recovery methods, and which recovery method works depends on the depth at which the goal is lost.

We also want to note that we are really excited about your engagement and your help in fine-tuning the paper, which we truly appreciate. It is great to see that you want to make it as good as it can be. So asking for changes again is not a problem at all, and we are very happy to make any further edits or changes that might still be needed. Thank you again for taking the time to help us make the paper easier to understand for RL practitioners, this is exactly who we want to reach.

Reviewer_L62UOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors explored the internal goal representation of misbehaved agents in goal generalization They designed 3 procgen environments with different types of ambiguity: Maze, CoinRun and AttributeMaze. In each task, misgeneralizing agents are compared with control agents that share the same structure and training recipe but are trained on a de-correlated distribution. They asked several questions:

  • whether goal information is linearly decodable
  • whether the policy is sensitive to goal edits
  • whether goal directed behavior can be recovered by activation steering or by relearning the downstream readout from frozen representations They found that seemingly same misgeneralizations have three distinct failures. Maze shows functional target substitution, CoinRun shows a policy shortcut, and Attribute Maze shows cue competition. The central claim is that where the goal is lost inside the agent determines which later repair works and thus agent steering should be done after careful internal diagnosis.
Critical strengths
  1. The authors went beyond just the behavior characterization of misgeneralization and linked it to progressive representation analysis, later steering of agents and their relationships across different environments.
  2. The careful experiment design, e.g., matched non-GM controls, random-initialization baselines, and per-checkpoint behavioral qualification, makes the claim systematic and clear.
  3. The activation-space Fisher information analysis reveals structure that Euclidean analyses miss and could be reused in other studies.
Critical weaknesses
  1. For the experiment design, the number of checkpoints used for analysis is limited. Consider the random nature of RL, the authors' claim could be stronger with larger scale analysis.
  2. In Maze and CoinRun, the steering direction is from world edits to which the GM policy barely responds, therefore, their failure largely follows from sec3.2 by design.
Questions for authors

I wonder if the diagnosis in this study can predict repair on checkpoints it was not derived from. I think this can be tested by independently training a few more cohorts in these environments and checking whether the diagnostics predict which repair succeeds before running it.

I also wonder, within one environment, do different checkpoints fall into different regimes? Since each environment currently shows one regime, the mechanism is confounded with the environment. A more clear approach is to decompose the potential environment-regime interaction by, for example, varying the training ambiguity within one environment.

2 comments

Thank you for the great review and the interesting questions. They led to two new appendix sections, and we greatly enjoyed thinking about them!

C1: Number of checkpoints, random nature of RL → Added three to five new seeds per environment and five environments.

  • More seeds. For every main-section environment we trained new GM agents with the same training code on unused seeds: three in Maze, PushToMarker and Attribute Maze, five in Coin. Before diagnosing them, we took the paper's findings as predictions for their depth and recovery. The numbers replicated and every new agent matched its prediction (Appendix F.1).
  • More environments. From three to eight, with four robot-arm tasks (ManiSkill3) and a quadruped task (Isaac Lab). Each depth and each failure type now appears in more than one environment, except the policy shortcut, which we induced on the robot arm only partly:
Target substitutionPolicy shortcutCue competition
Not readMaze (6), LocationReach (3)
Read, but not followedPushToMarker (6)Coin (8), (DirectionReach (3), DirectionHold (3))
Followed, but outcompetedAttribute Maze (6, plus 3 on a textured background), AttributeTable (3), SortZones (3), WalkToFlag (3)

C2: In Maze and Coin, steering comes from world edits the GM policy barely responds to → It is not an unfair comparison by design: Added explanations to why and a new probe-based steering variant, which replicates the result. You have an interesting and important point that should have been addressed. However, the experiment is not flawed but hits exactly what we want to show.

  • Why steering: We use steering to study the existing network and how it uses the information inside it, so we deliberately do not use oracle agents or structures from other networks. It is therefore not only a recovery attempt but also an analysis of what goal information is still present and can be amplified.
  • Maze. World edits barely move the policy, which indicates that there is no usable goal information to steer with. That is the depth this experiment should reveal, not a flaw of the design.
  • Coin. Here your premise does not hold: world edits do move the GM policy (KL ratio 0.1140.114, Table 1), so steering is informative, and it still fails.
  • New steering variant. To rule out that we used the wrong edit or cherry-picked the design, we added steering with a different information source, goal probes: along the ridge weights of a linear probe for the goal's position, and along its activation pattern, which accounts for nuisance objects. Both reproduce the same split by depth (Appendix E.1.3).
  • Conclusion: The same picture from the inside. In Maze, the goal cannot be decoded from the GM agents, so no reliable probe exists. As with the world edits, there is nothing to base steering on. In Coin, the goal can be decoded and the probe is reliable, yet steering still fails, because the policy does not act on the goal information we amplify. So steering fails for two different reasons: in Maze, there is no goal information to steer with, while in Coin the information is there but not used. Only where the policy follows the goal, as in Attribute Maze, does amplifying it work. The existing wiring cannot use the goal in Coin, which is why we then turn to rewiring.

Q1. Can the diagnosis predict repair on checkpoints it was not derived from? → Yes. This is exactly what we did, and every new agent matched (Appendix F.1).

Q2. Do different checkpoints within one environment fall into different regimes? → With the same training they share a depth, and the depth moves with the training.

  • One embodiment, all regimes. On the same robot arm we find all three depths and two of the three failure types, and induce the third partly (Appendix D.4.3). The regime depends on how the agent was trained, so it is no longer confounded with a single environment.
  • Same training. GM agents trained the same way land at the same depth (Appendix F.1), although how strongly they misgeneralize varies between them.
  • Varying training ambiguity. This suggestion triggered an ablation we really enjoyed (Appendix F.2). For one environment per depth, we increased how often goal and proxy are separated in training and followed the depth:
    • PushToMarker: the depth moves from read, but not followed, to followed. Reading the marker comes first, following later, and goal misgeneralization ends exactly where the policy starts to follow the goal.
    • LocationReach: a wider goal region moves the depth from not read to read, but not followed, and turns the proxy from a spot into a region.
    • Attribute Maze: showing yellow on both objects in 1/8 of the training environments already removes the competition. The policy turns toward the goal and wins the push and pull.

Thank you again for your review. We hope that our revisions now address your critiques and open questions, and we would be happy to answer any remaining questions before the discussion period ends!

Reviewer_Vv7UOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies goal misgeneralization, a failure mode where agents might learn a proxy goal that is optimal during training, but suboptimal at test time due to shifts in the environment distribution. The authors study what is the goal learned by the network when it goal misgeneralizes, and does so across 3 different environments. Three classifications for the failures are proposed: (i) target substitution, (ii) policy shortcut, and (iii) goal competition. The paper provides interventions that reduce goal misgeneralization for the latter 2 settings.

Critical strengths

Goal misgeneralization is an important safety failure that has been understudied in the field. I agree with the authors that a mechanistic view of the problem can be valuable and inform mitigations. I also think the experiments in the paper are potentially interesting and complement previous work well.

Critical weaknesses

Unfortunately, while I was looking forward to understanding the paper and gaining new insights from it, the current state of the writing made that impossible. The paper uses jargon that is not defined anywhere and is not standard in the literature. Mathematical notation is also just thrown in the paper without careful justification and appropriate introduction. This is also the case for the plot labels (e.g., what is tug of war? Where does that come from?). There are plenty of examples in the paper where new terms and definitions pop up from nowhere, making the paper really hard to follow.

Experiments are also limited to a narrow set of environments. I think a broader scope for the environments / proposed interventions would make the paper much convincing when trying to sustain some of its claims.

I encourage the authors to spend substantial effort rewriting the paper in a way that is understandable to a human reviewer. I am familiar with the area but had a very hard time reading it, and this hindered my understanding.

Questions for authors
  • You mentioned throughout the paper that behavior cannot distinguish the goal misgeneralization failures we saw in previous work (Di Langosco et al. (2022)). However, the 3 failures in that paper are indeed behavioral (location, direction, visual features), and they map one-to-one to the failures you present. Shall I thus understand your claim to be that you explain how these failures arise inside the network? I found that part confusing.
  • Assuming access to the true goal seems a demanding assumption for interventions (which in my understanding is what you need in the steering approach?). How could one circumvent that?
5 comments

Thank you for your review! We love that you were looking forward to reading and understanding the paper. That makes us all the more sorry that its clarity made this impossible. Your review triggered a full rewrite of the paper, which we hope is now much more readable and can give you the insights you were looking for in a clear way.

Clarity: undefined jargon, notation, plot labels → Addressed throughout, we rewrote large parts of the paper. We rewrote essentially all parts of the main text, with an additional focus on making it easy to skim. About 80% of the main text is new or rewritten, and all figures are redone.

  • Jargon. We added a glossary of every term we introduce (Appendix A). Additionally, we checked that every term and value in the main text is well defined, wherever possible also in the figure captions or on the axes. And "Tug of war" is gone.
  • Notation. The basics are introduced first, and the metrics are then built from them step by step. We polished and aligned all mathematical notation in the main text, and added a second glossary which lists the uncommon symbols (Appendix A).
  • Plot labels. We redid all plots to make them skimmable and understandable, so that any confusion can be resolved from the labels, legends and titles themselves, or as a last resort from the caption. Each caption now states the finding, and Table 1 now defines each measure and states which values set the depth.

We would be very happy to have any further feedback especially examples that help improve the clarity even further!

Narrow set of environments → Added five environments and three to five new seeds per environment.

  • Environments. We added four new robot-arm tasks (ManiSkill3) and a quadruped task (Isaac Lab), increasing the number of envrionments from 3 to 8. Moreover, goal misgeneralization had not been studied there, nor in robotics at a larger scale making our work the first to study goal misgeneralization systematically in robot control. Each depth and failure type now appears in more than one environment (except for the policy shortcut, which we could induce on the robot arm only partly, see Appendix F.3). GM seeds per environment, each with three Non-GM seeds:
Target substitutionPolicy shortcutCue competition
Not readMaze (6), LocationReach (3)
Read, but not followedPushToMarker (6)Coin (8)
Followed, but outcompetedAttribute Maze (6, plus 3 on a textured background), AttributeTable (3), SortZones (3), WalkToFlag (3)
  • Seeds. The counts include new GM agents on unused seeds. We predicted their depth and recovery before diagnosing them, and every new agent matched (Appendix F.1).

Q1. Langosco et al.'s failures are behavioral and map one to one to yours → We shifted the focus to classifying environments by depth, which explains recovery. The revision now separates two axes. The first is the depth (now the focus of the paper): (i) the policy does not read the goal, (ii) reads it but does not follow it, or (iii) follows it while the proxy outcompetes it. This taxonomy predicts which recovery works, which a behavioral classification by type of proxy can not (as far as we know). The second axis, the three failure types, is a secondary finding, similar to Langosco et al. but a richer grouping:

  • A target substitution replaces the goal with another target. In Maze this is not a memorized place but an extreme: when we open the walls, the agent goes to the new top-right corner (Appendix D.4.1).
  • A policy shortcut is a bias toward a subset of actions, not necessarily one direction.
  • Cue competition covers goal and proxy cues on one object or on two objects standing close together, which "visual features" does not capture, since most of our location proxies are visual too.

We want to stress, however, that the main focus and contribution of the paper is the depth taxonomy, and that the depth indicates which recovery works. The failure types are a second axis on top of it.

Q2. Access to the true goal seems demanding for interventions → Current interventions need it, and we leave this as open work. Steering, as used here, is a diagnostic that needs to know where the goal is, both with world edits and with the probe-based steering we added. Rewiring needs it less: goal labels, from a generalized teacher or from DAgger, are needed only once, while the new policy head is learned. After that, the rewired agent runs without any access to the goal, because its new head reads the goal from the existing network. We leave recovery without any goal labels as open work which we added to the discussion and and stated the use of the true goal more directly in the main text. We hope our work motivates new deployment-ready methods that use the depth finding to adjust their methods or understand why methods don't hold across all environments.

Thanks for the answer and the additional work.

I am happy to hear that my review incentivized a heavy revision of the paper, and thanks for the hard work. However, I think this is not the spirit of the rebuttal period, in my understanding! If the paper changed significantly (more than 80%, as you said), it would require another detailed review for me to assess the quality of the work again (which takes hours when you want to do it carefully :) ). That is equivalent to a full second submission, which would be unfair to the review process.

I also want to emphasize that in its first version, the paper was really, really hard to read and much below the standard of publications for an academic paper at a top ML conference.

I think this work has potential, but I suggest that the authors resubmit the paper to a future venue. Best of luck!

Thank you for the quick reply. We fully understand that our reply sounded like it would require another full review, which is definitely not in the scope of a rebuttal! Saying that 80% of the sentences were rewritten sounds like 80% of the content is new, but this is not the case. The content is the same, but the clarity and the way it is written were carefully redone.

In your review you wrote: “I encourage the authors to spend substantial effort rewriting the paper in a way that is understandable to a human reviewer.” This is exactly what we did. The content, the experiment designs, the scientific arguments and the results of the paper have not changed. They will not require a new review and will not be new to you, but will hopefully be understood immediately on a new read. The rewrite aims to convey the main message and the main findings much more directly, delivering the clarity it lacked before. The only addition to the main section in content is one new environment, PushToMarker. The other new environments and the new seeds are in the appendix and only replicate the same findings, in response to your concern about the narrow set of environments.

So reviewing this revision does not mean re-evaluating a new paper or new methods, but checking whether the same content can now be read and understood better. The abstract, introduction, Table 1 and the figure captions show this in also in short time (now per design!).

We put a lot of effort into the rebuttal phase to meet your request, and we would be very happy if you could consider the revision for this conference. We believe a full review is not necessary, as the improvement in clarity should be immediately visible. Even if you decide not to review it, we would be grateful if you could tell us whether the revision addresses the concern you raised. Thank you very much for your help and your review of our paper!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and title change request

We thank all three reviewers for their rich and high-quality reviews, and especially for everyone mentioning how important this topic is. It is great to see it getting recognition! The paper is very long and has a lot of details, but the reviews helped a lot to see where it was insufficient. We used every point to rewrite the paper, and we are much happier with it and prouder of it now. Thank you again for all the comments!

Moreover, since the revision now centers the paper on where the goal is lost, we would like to request a new title: “Where the Goal Gets Lost: The Depth of Goal Misgeneralization Predicts Its Recovery”. We keep the original title for now and would change it only with the agreement of the reviewers and the area chair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.