Behaviorally Silent Now, Causally Active Later: Deferred State Effects in Recurrent Policies for Reinforcement Learning

Observation perturbations in reinforcement learning (RL) agents with recurrent policies can have lasting effects on both the policy's internal state and its subsequent behavior. However, when such delayed effects occur, it remains unclear whether they arise from the immediate action change induced by the perturbation or from perturbation information retained in the policy's internal state. To address this, we introduce current action equivalence, which separates these pathways by comparing internal states that produce the same current action. For the one-shot intervention on feedforward policies, reproducing the current action reproduces the downstream effect. In recurrent policies, however, real observation perturbations can induce internal state changes that remain behaviorally silent at the current step yet become increasingly expressed through future actions. We identify these changes as deferred state effects and show through interventions that they causally influence future actions, closed-loop trajectories, and task reward. The effect is consistent across perturbations, recurrent architectures, environments, and independently trained policies. These results show that the effects of a perturbation can remain behaviorally silent at the present step while continuing to shape future decisions in recurrent policies.