- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how observation perturbations affect the future behavior of recurrent policies. The authors first introduce current action equivalence to separate the immediate effect on the current action from the delayed effect kept in the recurrent state. They then split the perturbation-induced change in the hidden state into three parts: one that shows up in the current action, a deferred part that doesn't affect the current action but strongly affects future actions, and a remaining part. By restoring each part with the same norm, they show that restoring the deferred part brings future actions, trajectories, and reward much closer to the unperturbed run than restoring the remaining part.
Critical strengths
- The question is clear and interesting, and current-action equivalence is a clean way to separate the action-mediated path and the state-retained path.
- The results are consistent across recurrent architectures, environments, and perturbation types.
Critical weaknesses
- The main results are partly expected by construction. For example, the deferred subspace is defined as the null space directions with the largest future-action sensitivity; meanwhile, the remaining part is defined as the low-sensitivity part. It is therefore expected in table 2 that restoring the deferred part recovers future behavior much better. A fairer control could be to restore the perturbation along a random null-space subspace of the same dimension.
- This work is not well positioned against prior works. For example, the idea that activity in the output-null space can shape future outputs is well established in neuroscience (e.g. Kaufman et al., 2014). Moreover, without discussion on why it’s relevant, it’s hard to see the significance or practical use of this work, e.g., for detecting adversarial perturbations or improving robustness.
Questions for authors
- Related to Weakness 1 above, how does restoring the perturbation along a random null-space subspace of the same dimension compare with the deferred intervention?
- How do the authors think the deferred components could be used in practice?
- Can the authors discuss whether this framework can be transferred to stochastic policies, where exact current action equivalence is not well defined?
- Can the authors comment on the scalability of this method to much larger state and action spaces, given that the method requires computing the Jacobian over H future steps and an eigendecomposition in the hidden-state dimension?
Response to Question 4 How does the method scale to larger state and action spaces?
We implemented projected matrix-vector products using JVP/VJP and iterative leading-eigenspace estimation, avoiding explicit future Jacobians and dense state-space Gram matrices. On all 12 native policies at H10/H20/H50, projector and paired-correction errors relative to the explicit construction are below 10−6. Isolated measurements use one worker per GPU, warmup, and the median of three repetitions. Mean explicit/iterative times are 0.143/0.393 seconds at H10, 0.468/0.716 at H20, and 2.589/1.588 at H50, so iteration helps at H50 but is slower at shorter horizons. Peak allocated memory is similar at the tested native state dimension. Both methods use the same backend and a fixed rank from the explicit 90%-trace reference.
Overall, these results demonstrate that the proposed matrix-free implementation preserves the accuracy of the explicit construction on the tested policies and offers computational advantages at longer horizons. Evaluating its scalability to substantially larger hidden-state dimensions and developing fully iterative trace/rank selection remain important directions for future work.