acceptodds
Under review as a conference paper at ICLR 2027

Behaviorally Silent Now, Causally Active Later: Deferred State Effects in Recurrent Policies for Reinforcement Learning

Abstract

Observation perturbations in reinforcement learning (RL) agents with recurrent policies can have lasting effects on both the policy's internal state and its subsequent behavior. However, when such delayed effects occur, it remains unclear whether they arise from the immediate action change induced by the perturbation or from perturbation information retained in the policy's internal state. To address this, we introduce current action equivalence, which separates these pathways by comparing internal states that produce the same current action. For the one-shot intervention on feedforward policies, reproducing the current action reproduces the downstream effect. In recurrent policies, however, real observation perturbations can induce internal state changes that remain behaviorally silent at the current step yet become increasingly expressed through future actions. We identify these changes as deferred state effects and show through interventions that they causally influence future actions, closed-loop trajectories, and task reward. The effect is consistent across perturbations, recurrent architectures, environments, and independently trained policies. These results show that the effects of a perturbation can remain behaviorally silent at the present step while continuing to shape future decisions in recurrent policies.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement learning, Recurrent policy, Causal interventions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pNSXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how observation perturbations affect the future behavior of recurrent policies. The authors first introduce current action equivalence to separate the immediate effect on the current action from the delayed effect kept in the recurrent state. They then split the perturbation-induced change in the hidden state into three parts: one that shows up in the current action, a deferred part that doesn't affect the current action but strongly affects future actions, and a remaining part. By restoring each part with the same norm, they show that restoring the deferred part brings future actions, trajectories, and reward much closer to the unperturbed run than restoring the remaining part.

Critical strengths
  1. The question is clear and interesting, and current-action equivalence is a clean way to separate the action-mediated path and the state-retained path.
  2. The results are consistent across recurrent architectures, environments, and perturbation types.
Critical weaknesses
  1. The main results are partly expected by construction. For example, the deferred subspace is defined as the null space directions with the largest future-action sensitivity; meanwhile, the remaining part is defined as the low-sensitivity part. It is therefore expected in table 2 that restoring the deferred part recovers future behavior much better. A fairer control could be to restore the perturbation along a random null-space subspace of the same dimension.
  2. This work is not well positioned against prior works. For example, the idea that activity in the output-null space can shape future outputs is well established in neuroscience (e.g. Kaufman et al., 2014). Moreover, without discussion on why it’s relevant, it’s hard to see the significance or practical use of this work, e.g., for detecting adversarial perturbations or improving robustness.
Questions for authors
  1. Related to Weakness 1 above, how does restoring the perturbation along a random null-space subspace of the same dimension compare with the deferred intervention?
  2. How do the authors think the deferred components could be used in practice?
  3. Can the authors discuss whether this framework can be transferred to stochastic policies, where exact current action equivalence is not well defined?
  4. Can the authors comment on the scalability of this method to much larger state and action spaces, given that the method requires computing the Jacobian over H future steps and an eigendecomposition in the hidden-state dimension?
1 comment
AuthorsOpenReview ↗

Response to Question 4 How does the method scale to larger state and action spaces?

We implemented projected matrix-vector products using JVP/VJP and iterative leading-eigenspace estimation, avoiding explicit future Jacobians and dense state-space Gram matrices. On all 12 native policies at H10/H20/H50, projector and paired-correction errors relative to the explicit construction are below 10−610^{-6}. Isolated measurements use one worker per GPU, warmup, and the median of three repetitions. Mean explicit/iterative times are 0.143/0.393 seconds at H10, 0.468/0.716 at H20, and 2.589/1.588 at H50, so iteration helps at H50 but is slower at shorter horizons. Peak allocated memory is similar at the tested native state dimension. Both methods use the same backend and a fixed rank from the explicit 90%-trace reference.

Overall, these results demonstrate that the proposed matrix-free implementation preserves the accuracy of the explicit construction on the tested policies and offers computational advantages at longer horizons. Evaluating its scalability to substantially larger hidden-state dimensions and developing fully iterative trace/rank selection remain important directions for future work.

Reviewer_6fHjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies how observation perturbations affect the future behavior of recurrent RL policies beyond their effects on the current action. The authors introduce current-action equivalence and construct a finite-horizon Gramian of future-action Jacobians restricted to the current-action null space. Its leading eigenspace is used to decompose perturbation-induced internal-state differences into deferred and remaining components. Through matched interventions across MuJoCo environments and recurrent architectures, the authors show that restoring the deferred component toward a clean reference recovers future actions and closed-loop trajectories more effectively. Additional experiments examine directional specificity, robustness to construction choices, and replication across independently trained policies.

Critical strengths

The paper addresses an interesting question about the delayed effects of observation perturbations on recurrent policies. The presentation is clear, particularly the distinction between effects mediated by the current action and those retained in the internal state. The mathematical construction provides an interpretable way to identify directions with different temporal sensitivities.

Also, the empirical evaluation supports the analysis through carefully controlled interventions, including matched intervention norms, current-action matching, and directional controls. Closed-loop evaluations and experiments across recurrent architectures provide further evidence for the reported phenomenon.

Critical weaknesses

The implications for downstream applications remain unclear. Although the controlled restoration experiments are informative, they rely on access to a clean reference state that would generally be unavailable during deployment. The paper does not sufficiently explain how the identified subspace could guide practical detection, prevention, or correction of undesirable behavior. A clearer connection between the empirical findings and actionable improvements to recurrent agents would strengthen the significance of the contribution.

Questions for authors
  1. The restoration experiments use the difference between clean and perturbed internal states to determine how to intervene. How could the proposed analysis guide practical behavioral correction when a clean reference is unavailable?

  2. Could the authors provide a deeper interpretation of what the deferred directions represent and why restoring their coordinates improves subsequent behavior? For example, visualizing the spectrum of GdeferG_{\mathrm{defer}}, and the clean–perturbed displacement along its leading directions. Can these directions be associated with identifiable aspects of the agent’s memory or behavior, beyond their definition through future-action sensitivity?

  3. Could the authors clarify the relationship between their construction and gradient outer-product approaches to sufficient dimension reduction, including the smoothed-gradient approach of Yuan et al. (2025) and the AGOP-based central-subspace recovery results of Zhu et al. (2026)? The AGOP perspective on feature learning developed in the Neural Feature Ansatz may also provide useful context (Radhakrishnan et al., 2024).

Reviewer_J4ehOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose in the paper a methodology for studying the effect of noisy observations on RNN.. They propose a methodology for trying to understand whether the noise affects the current action only or whether it is is stored in the internal state and affects subsequent actions. They also propose simulation results.

Critical strengths

The paper us relatively well written and clear.

Critical weaknesses

Lack of clear problem statement. You really need for example to better characterize your noise.

The problem statement is also not well motivated.

Try to use more maths to characterize what you are doing!

Questions for authors

Why didn't you use a more theoretical approach at first? I think it is the right way to study your problem.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.