What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization

Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. From the outside these failures look alike, yet we still understand little about what happens to the intended goal inside misgeneralized agents. We trace the goal through agents in three reinforcement-learning environments, asking whether it is still represented, whether the policy reads it out, and what organizes behavior instead. We find three distinct failures of policy alignment, in which the goal gets lost at different depths: (1) functional target substitution, where the goal is inert to the policy and behavior follows an alternative target, (2) a policy shortcut, where the goal is represented but not used as a goal and behavior follows a learned action rule, and (3) cue competition, where the goal reaches the decision but is outweighed by a proxy cue. Where the goal gets lost explains which recovery works. Under cue competition, steering along the axis on which the two cues compete raises intended-goal reach from $16.0\%$ to $95.3\%$, while the shortcut recovers only after learning a new readout, from $31.6\%$ to $84.3\%$, and under target substitution neither restores goal-directed behavior. So goal misgeneralization is not one failure, and recovering the goal requires knowing where it got lost.