Building State v.s. Using State: Representation and Utilization in Multi-Step Tasks

Multi-step tasks require a model to do two things well: maintain the state needed for future decisions and use that state when producing actions or answers. We study these problems separately in language models and partially observed reinforcement learning agents. In a controlled maze, map-related information becomes linearly decodable before strong task performance, and matched belief swaps show that this internal map content affects behavior. Yet a representation-targeted auxiliary that raises decodability leaves tight-horizon success essentially unchanged (0.093 to 0.094), while policy-level supervision raises it to 0.497. Craftax shows the same asymmetry at larger scale: recurrent state is necessary for strong performance and contains hidden spatial information, but substantially increasing position decodability does not improve reward. We then ask whether this diagnosis can be used to improve the models. It can. After informative state has formed, interventions aimed at the state-to-behavior mapping improve downstream performance in both domains. Post-convergence efficiency pressure raises Craftax score from 15.09 to 16.56, and latent-guided distillation raises low-data answer accuracy from 0.86 to 0.94 in the language setting. These results suggest that multi-step learning can be limited by state utilization even after useful internal state has already been learned.