acceptodds
Under review as a conference paper at ICLR 2027

GAMPO: Auditing and Optimizing LSTM Memory for Partially Observable Reinforcement Learning

Abstract

Recurrent policies must preserve useful evidence while limiting sensitivity to nuisance history. Forget-gate products omit paths through coupled LSTM cell–hidden dynamics. We use the complete joint-state Jacobian to bound finite history perturbations and define a full-path memory tail. Filter contraction and decoder calibration translate this tail into belief and control errors; a terminal-decision refinement avoids unnecessary horizon factors in recall tasks. Gate-Aware Memory Policy Optimization (GAMPO) combines an exponential teacher with sensitivity, information-state, and projection losses. Its population regret decomposition separates representation, critic, and projection errors, with explicit conditions linking fitted losses to these terms. Four controlled POMDPs reveal substantial limitations: the reported model-free variant remains near chance on recall tasks and can underperform unregularized sequence baselines. Full-path influence improves over a forget-only proxy on active sensing, and an independent fast-refresh control shows a regret benefit over omission. Short-delay and exact-belief controls expose persistent learning limitations even when memory or representation demands are reduced. These findings support full-path sensitivity as an auditing tool and expose the additional requirements for effective memory regularization.

Primary area:
reinforcement learning
Keywords:
Partially observable reinforcement learning, recurrent reinforcement learning, LSTM memory, policy optimization, memory auditing, approximate information states

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tv8MOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to quantify and regularize memory in recurrent policies for partially observable reinforcement learning. It uses the hidden-state LSTM Jacobian to bound historical input influence, connects these bounds to belief and control errors under additional assumptions, and proposes GAMPO, which combines policy mirror descent with sensitivity and information-state losses. In the experiment, they evaluate GAMPO on four synthetic POMDP tasks: delayed recall, conditional recall, a filter-stable chain, and active sensing. Also, using baseline comparisons, regularization ablations, and diagnostic experiments on memory sensitivity and representation quality.

Critical strengths

The paper makes an interesting distinction between being sensitive to past inputs and retaining information useful for decisions. By considering both the cell and hidden states, the analysis captures effects that forget gates alone miss. This helps explain why reducing sensitivity to past inputs can remove distractions or erase important clues.

Critical weaknesses

The theoretical assumptions seems strong and limit where the results apply. The main connection between memory and belief accuracy requires filter contraction, along with accurate decoding and Lipschitz conditions. If I understand correctly, the contraction assumption does not hold in the paper’s delayed-recall and active-sensing tasks. Although some results apply more broadly, this central part of the theory does not cover several important tasks. Also, the theory does not show that the proposed training method meets these assumptions.

In experiment, as authors acknowledged, GAMPO-P performs worse than simpler baselines on other tasks, and its benefit occurs when past information is unnecessary, so it does not demonstrate improved retention of useful memory.

Questions for authors

Can the theoretical assumptions be weakened? In particular, can you obtain useful bounds for delayed recall and active sensing, where uniform filter contraction does not hold?

What causes the poor recall performance of GAMPO-P?

2 comments

Thank you for these precise questions. Both assumptions can be weakened to cover delayed recall and active sensing, and new experiments identify what limits recall. GAMPO-P and GAMPO-KF are our methods; the other encoders of Table 1 are baselines.

1. Path-wise stability instead of uniform contraction. Let eh=TV(b^h,bh)e_h=\mathrm{TV}(\hat b_h,b_h) with the one-step defects δh\delta_h of Eq. (3). If along the realized history the Bayes update is κh\kappa_h-Lipschitz in TV on a domain containing the compared beliefs, the recursion of Theorem 5 gives

eh≤min⁡{1,e1∏j=2hκj+∑s=2hδs∏j=s+1hκj}.e_h\le\min\Big\lbrace1,e_1\prod_{j=2}^h\kappa_j+\sum_{s=2}^h\delta_s\prod_{j=s+1}^h\kappa_j\Big\rbrace.

Theorem 5 is the case κh≤ρB<1\kappa_h\le\rho_B<1. In delayed recall every post-cue update is the identity (κh=1\kappa_h=1), so eH≤min⁡{1,e1+∑s≥2δs}e_H\le\min\lbrace1,e_1+\sum_{s\ge2}\delta_s\rbrace; in E2 the short-cue reveal (C=0C=0) has κ=0\kappa=0 and resets the error. Accurate cue decoding and small one-step defects thus suffice, with no forgetting required.

2. Active sensing in log-odds. In E4 a noisy probe can expand TV (by up to (1−q)/q(1-q)/q), so we work with ℓh=logit⁡bh=ℓh−1+λh\ell_h=\operatorname{logit}b_h=\ell_{h-1}+\lambda_h, where λh=0\lambda_h=0 after a skip and ±log⁡1−qq\pm\log\frac{1-q}{q} after a probe. For interior decoded beliefs, set rh=ℓ^h−ℓ^h−1−λhr_h=\hat\ell_h-\hat\ell_{h-1}-\lambda_h and Dh=∣ℓ^1−ℓ1∣+∑s=2h∣rs∣D_h=|\hat\ell_1-\ell_1|+\sum_{s=2}^h|r_s|; telescoping and ∣σ(u)−σ(v)∣≤tanh⁡(∣u−v∣/4)|\sigma(u)-\sigma(v)|\le\tanh(|u-v|/4) give

eh≤tanh⁡(Dh/4)≤Dh/4,e_h\le\tanh(D_h/4)\le D_h/4,

exact at zero defect and free of contraction.

3. Control without a Lipschitz policy head. Each Qh⋆(⋅,a)Q^\star_h(\cdot,a) is a supremum of linear functionals of the belief with range [0,Bh][0,B_h], BhB_h the maximal remaining return, hence BhB_h-Lipschitz in TV. If the action is ϵh\epsilon_h-greedy for Qh⋆(b^h,⋅)Q^\star_h(\hat b_h,\cdot), two belief substitutions and the performance-difference identity give

V⋆−Vπ^≤∑h=1HEdhπ^[2Bheh+ϵh].V^\star-V^{\hat\pi}\le\sum_{h=1}^H\mathbb E_{d_h^{\hat\pi}}\big[2B_he_h+\epsilon_h\big].

For the binary terminal decisions of E1–E3 a margin argument is sharper: the greedy label is Bayes-optimal whenever eH<∣bH−12∣e_H<|b_H-\tfrac12|, so V⋆−Vπ^≤E[∣2bH−1∣1{eH≥∣bH−12∣}]V^\star-V^{\hat\pi}\le\mathbb E\big[|2b_H-1|\mathbf 1\lbrace e_H\ge|b_H-\tfrac12|\rbrace\big]. In E1, ∣bH−12∣=0.4|b_H-\tfrac12|=0.4: any decoded belief within 0.4 of the truth recalls optimally. In E4, Bh≤1.05B_h\le1.05 and Section 2 supplies ehe_h. Both are checkable on trained models: on five retrained GAMPO-KF policies for E1, the decoded terminal belief stays within 0.063 of the exact posterior on all 10,000 held-out histories per seed, so the margin holds everywhere; the path-wise bound e1+∑sδse_1+\sum_s\delta_s averages 0.094, and Emin⁡{1,2eH+ϵH}=0.114\mathbb E\min\lbrace1,2e_H+\epsilon_H\rbrace=0.114 bounds the observed gap of 0.026.

4. What limits recall. The influence penalty is not the cause: omitting it changes GAMPO-P's E1/E2 regret by 0.5 [−2.7, 3.5] and −4.2 [−26.9, 15.5], every encoder baseline also stays at chance, and GAMPO-P learns recall at delay 8 (0.876; optimum 0.900). In E1 the only cue-dependent signal is the terminal reward (Proposition 21), which must cross 64 transitions. A new initialization audit (20 seeds) measures this transport: the cue's ∥∂ZH/∂X1∥\Vert\partial Z_H/\partial X_1\Vert falls from 0.14 at delay 8 to 0.034 at 16 and 6×10−66\times10^{-6} at 64, and the shared critic's cue sensitivity at delay 64 is 5×10−95\times10^{-9}. Theorem 20, extended to cue noise qq, gives Vπ≤12+(12−q)εˉV^\pi\le\frac12+(\frac12-q)\bar\varepsilon for the paired cue-flip change εˉ\bar\varepsilon of the terminal action, tight in our runs (εˉ<10−7\bar\varepsilon<10^{-7}, value 0.500 for GAMPO-P). Accordingly, two diagnostic controls (exact-belief critic, chrono-initialized gates) let the cue through in only 2 of 20 runs, whereas GAMPO-KF, whose consistency loss supervises the belief at every step, learns the 64-step recall in all 5 (εˉ=0.94\bar\varepsilon=0.94, value 0.874). On E2, an exact-belief critic lifts the four-step branch from 0.51 to 0.65–0.78 while the 64-step branch stays at chance, separating a critic bottleneck from the transport one.

5. The loss and useful memory. On E3, where 75% of observations are erased and history is needed, we restricted GAMPO-P's influence loss to inputs older than a window mm, as Corollary 6 prescribes (20 paired seeds). With m=16m=16, regret is 959 versus 970 without the loss and the value after an erased terminal observation is 0.748 in both, against 0.663 for the all-history loss; with m=8m=8, inputs older than the window lose 2–4-fold of their influence while the last seven keep theirs. With the prespecified fast-refresh confirmation (regret lowered by 300.8 [272.2, 327.4]), these results show that the loss removes stale history without erasing decision-relevant memory once its window follows the theory.

Revisions. We will add the path-wise, log-odds and margin bounds to Section 4 (proofs in Appendix C.4), and the recall diagnosis and windowed-loss experiments to Section 7, marking ours versus baselines in every table.

I thank the authors for their response. Although the response partially addresses my concerns, I am still not convinced that the contribution meets the bar for acceptance. Accordingly, I will keep my score (including confidence).

Reviewer_4QZSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory retention and historical sensitivity in LSTM for partially observable reinforcement learning. The authors argue that forget-gate products do not fully characterize LSTM memory, since it is coupled with cell–hidden dynamics. They leverage the full Jacobian to measure the historical influence and provide theoretical bounds for memory sensitivity to belief approximation and control errors under additional assumptions. The authors further introduce Gate-Aware Memory Policy Optimization (GAMPO), which incorporates sensitivity regularization and auxiliary information-state objectives to a policy mirror descent framework. Numerical results on four synthetic POMDP environments show that full-path sensitivity provides a more informative diagnostic than forget-gate-based measures.

Critical strengths

The paper provides a full-Jacobian-based approach to characterize LSTM historical sensitivity, accounting for the coupled cell–hidden dynamics overlooked by forget-gate products. The authors also provide theoretical bounds connecting historical input perturbations to hidden sensitivity, belief approximation error, and control performance. Numerical experiments further show that full-path Jacobian sensitivity better describes the effects of historical input perturbations than forget-gate-based measures, supporting its usefulness as a diagnostic tool.

Critical weaknesses
  1. Weak connection between the theoretical analysis and the proposed regularization Although the paper provides theoretical bounds characterizing how historical input perturbations affect the belief approximation and control performance, the connection between these theoretical results and the proposed sensitivity regularization is not sufficiently established. First, the theory focuses on recurrent-state sensitivity ∂Zh∂Xs\frac{\partial Z_h}{\partial X_s}, but the implemented regularization penalizes policy-logit sensitivity ∂ℓh∂Xs\frac{\partial \ell_h}{\partial X_s}, which does not directly control the memory tail. Second, minimizing historical sensitivity does not necessarily improve control performance, as task-relevant information may also be suppressed. It is still unclear how the proposed regularization is justified by the sensitivity analysis.

  2. Insufficient empirical evidence that the regularization is effective In Table 1, GAMPO-P performs similarly to existing methods on both E1 and E2 and underperforms on E3 and E4. Furthermore, Table 2 shows that removing the sensitivity regularization significantly improves performance on E3, while its effect on E4 cumulative regret is not statistically significant. Although GAMPO-KF achieves better results on some tasks, it relies on a known Bayesian filtering mechanism, making it difficult to identify whether these improvements are from the proposed regularization or not. Therefore, the experiments provide limited evidence that the proposed method improves task-relevant memory learning.

Questions for authors

Although the authors provide additional analyses of the negative experimental results in the appendix, the underlying causes remain unclear. Can the authors further explain why GAMPO-P fails on E1 and E2, as well as why sensitivity regularization degrades performance on E3?

3 comments

Thank you for pinpointing the step from the sensitivity bound to the training objective. We make this link explicit, show that the theory prescribes which history to penalize, and test it in new runs. Throughout, GAMPO-P and GAMPO-KF are our methods; LSTM-, GRU-, Transformer-, S4D- and MLP-PMD are encoder baselines.

1. Why the penalty acts on the policy, and on which history. Control depends on the action law, which the state bound controls. With Lℓ,hL_{\ell,h} bounding the readout derivative, the chain rule gives ∥∂ℓh/∂Xs∥≤Lℓ,hΓh,s\|\partial\ell_h/\partial X_s\|\le L_{\ell,h}\Gamma_{h,s}. Since TV(softmax u,softmax v)≤∥u−v∥2/(22)\mathrm{TV}(\mathrm{softmax}\,u,\mathrm{softmax}\,v)\le\|u-v\|_2/(2\sqrt2), two histories that share their last mm inputs satisfy

TV{πh(⋅∣Hh),πh(⋅∣Hh′)}≤DX22 Thπ(m)≤DXLℓ,h22 Th(m),Thπ(m)=∑s≤h−msup⁡∥∂ℓh∂Xs∥.\mathrm{TV}\{\pi_h(\cdot\mid H_h),\pi_h(\cdot\mid H_h')\}\le\frac{D_X}{2\sqrt2}\,T^\pi_h(m)\le\frac{D_XL_{\ell,h}}{2\sqrt2}\,T_h(m),\qquad T^\pi_h(m)=\sum_{s\le h-m}\sup\Big\|\frac{\partial\ell_h}{\partial X_s}\Big\|.

The state tail Th(m)T_h(m) certifies this policy tail, and our influence loss is a sampled surrogate of ThπT^\pi_h: it restricts only history that reaches the decision, leaving the recurrent state free for the predictive heads. Corollary 6 then states what may be suppressed: in TV(b^h,bh)≤ϵdec+CBρBm+LD,TVDXTh(m)\mathrm{TV}(\hat b_h,b_h)\le\epsilon_{\rm dec}+C_B\rho_B^m+L_{D,\rm TV}D_XT_h(m), only the tail beyond a window mm that the filter itself forgets, since suppressing evidence inside the window raises ϵdec\epsilon_{\rm dec}. The submitted loss sums over all earlier inputs, i.e. m=1m=1.

2. E3: cause of the degradation and a theory-guided fix. On E3 (ρB=0.9\rho_B=0.9, 75% of observations erased), a policy that sees only its last mm inputs attains at most 0.625, 0.805, 0.868 and 0.884 for m=1,4,8,16m=1,4,8,16 (Bayes 0.885), so the m=1m=1 loss suppresses evidence that is still informative. We restricted GAMPO-P's loss (λΓ=0.1\lambda_\Gamma=0.1) to Th(m)T_h(m) in 20 new paired seeds:

GAMPO-P (ours)λΓ=0\lambda_\Gamma=0m=1m=1 (submitted)m=4m=4m=8m=8m=16m=16
Regret97011581058981959
Final value0.7870.7310.7510.7830.788
Value, last obs. erased0.7480.6630.6860.7350.748

Relative to λΓ=0\lambda_\Gamma=0, regret changes by +187 [165, 209], +87 [67, 106], +11 [−7, 28] and −11 [−28, 5] as mm grows: the degradation disappears once the window covers the evidence the filter retains. The policy's input gradients show why: the m=1m=1 loss more than halves the dependence on the last seven inputs (0.067 vs 0.153), exactly the evidence an erased observation requires; at m=8m=8 it is intact (0.166), while the dependence on older inputs falls 2–4-fold.

3. Why E1 and E2 fail. The penalty is not the bottleneck: every encoder baseline also stays at chance, and omitting it changes GAMPO-P's regret by only 0.5 [−2.7, 3.5] and −4.2 [−26.9, 15.5]. In E1 only the terminal reward depends on the cue (cf. Proposition 21), so the signal that must teach the LSTM to keep it crosses 64 transitions. A new initialization audit (20 seeds) measures this path: the cue's ∥∂ZH/∂X1∥\|\partial Z_H/\partial X_1\| is 0.14, 0.034 and 6×10−66\times10^{-6} at delays 8, 16 and 64, matching where GAMPO-P learns (0.876; optimum 0.900), becomes unstable and fails, and the shared critic has cue sensitivity 5×10−95\times10^{-9} at delay 64. Theorem 20, extended to cue noise qq, gives Vπ≤12+(12−q)εˉV^\pi\le\frac12+(\frac12-q)\bar\varepsilon, where εˉ\bar\varepsilon is the paired cue-flip change of the terminal action law. The bound is tight in our runs (GAMPO-P: εˉ<10−7\bar\varepsilon<10^{-7}, value 0.500; GAMPO-KF: εˉ=0.94\bar\varepsilon=0.94, value 0.874), and the gradient that would enlarge εˉ\bar\varepsilon is built from the same Jacobian products. In new runs (5 seeds each), two diagnostic controls (an exact-belief critic, chrono-initialized gates) let the cue through only rarely (2 of 20), whereas GAMPO-KF, whose consistency loss supervises the belief at every step, learns the 64-step recall in 5 of 5. E2's 64-step branch fails for the same reason; its four-step branch is limited by the learned critic, as an exact-belief critic raises its accuracy from 0.51 to 0.65, and to 0.78 without the m=1m=1 loss, which also penalizes this cue. GAMPO-KF reaches 0.92/0.95 on the two branches.

4. Evidence that the regularizer helps. In the prespecified fast-refresh confirmation (20 new paired seeds), the loss lowers GAMPO-P's regret by 300.8 [272.2, 327.4] and raises final value by 0.105. On E4, replacing the full-path Jacobian by the forget-gate proxy raises regret by 503.6 [470.9, 538.0]: the full cell–hidden coupling matters on this memory-demanding task. GAMPO-KF, our model-informed variant, uses the known Bayes operator and shows what the same actor and update achieve with an exact information-state target.

Revisions. We will add the policy-tail bound and window rule to Section 5, the windowed-loss and recall experiments to Section 7, and mark ours versus baselines in every table.

Thanks for the author's clarification on the theoretical link between recurrent-state sensitivity and the policy-logit penalty. I have one more question. The E3 results also show that a small window can suppress useful information. On a new task where the relevant memory horizon is unknown, how would you choose mm in practice? Is there a practical section rule?

Thank you for the follow-up, and we are glad the clarification was helpful. Yes, there is a practical rule: the window can be read off a short truncation test on the policy itself, without the model or the memory horizon. We fixed the rule before any run and tested it in 90 new runs. It recovers the best windows on E3 and in the fast-refresh regime, and on a new task it selects the window with the lowest regret.

1. What the theory asks of mm. Corollary 8 (Corollary 6 of the submission) allows suppressing the tail beyond mm once the evidence older than mm, bounded by CBρBmC_B\rho_B^m, is negligible. The two possible errors are asymmetric. A window that is too short suppresses evidence the filter still uses (E3, m=1m=1: regret +187), whereas one that is too long only weakens the loss towards λΓ=0\lambda_\Gamma=0, since at m≥Hm\ge H nothing is penalized. A practical rule should therefore measure how much value the policy draws from inputs older than mm, and round up.

2. Selection rule. Train a pilot policy without the loss and freeze it. On fresh episodes, replace every input older than the last mm by the input at the same position of an independent episode (this keeps the inputs in distribution but removes their information), and record the value lost, ΔV(m)=V−Vm\Delta V(m)=V-V_m. Choose the shortest window on a doubling grid that keeps 95% of the value the policy draws from history,

m^=min⁡{m∈{1,2,4,8,16}:ΔV(m)≤0.05ΔV(1)},\hat m=\min\lbrace m\in\lbrace1,2,4,8,16\rbrace:\Delta V(m)\le0.05\Delta V(1)\rbrace,

and m^=1\hat m=1 when ΔV(1)≤0.01\Delta V(1)\le0.01, i.e. when history carries no usable value. ΔV(m)\Delta V(m) is the measured counterpart of CBρBmC_B\rho_B^m: for the Bayes policy on E3 it decays geometrically, as (ζρ)m(\zeta\rho)^m with erasure probability ζ\zeta. The test needs only forward passes of the frozen policy and returns (in the simulator we average the exact expected terminal reward, which only lowers the variance), and it applies unchanged to any memory architecture. The grid and both thresholds were fixed in advance and not tuned.

3. Validation (protocol fixed in advance, every run reported). Pilot: GAMPO-P without the loss, 10 new seeds per setting, test after 25 of 100 updates and on the final policy (10,000 histories).

SettingBest windowsm^\hat m after 25 updatesm^\hat m, final policy
E38, 1616 in 10/10 seeds16 in 7/10, 8 in 3/10
Fast refresh11 in 10/101 in 10/10
E3S (new)8, 16 (below)8 in 10/108 in 10/10

On E3, the measured profile ΔV(m)/ΔV(1)=0.77,0.38,0.06,0.00\Delta V(m)/\Delta V(1)=0.77,0.38,0.06,0.00 for m=2,4,8,16m=2,4,8,16 closely tracks the Bayes profile 0.68,0.31,0.06,0.000.68,0.31,0.06,0.00, and every selected window removes the cost of the loss (Table 1). In the fast-refresh regime, ΔV(1)≤0.003\Delta V(1)\le0.003 in every seed, and the rule returns m=1m=1, where the loss lowers regret by 300.8 [272.2, 327.4].

E3S (ρ=0.8\rho=0.8, 40% of observations erased, 64 distractor coordinates) was not used before. On 10 new paired seeds with λΓ=0.1\lambda_\Gamma=0.1, the selected window gives the lowest regret on the grid, −15.1 [−27.8, −3.2] relative to λΓ=0\lambda_\Gamma=0, and keeps the final value (−0.001 [−0.006, 0.003]). Shorter windows lower the final value, which is the suppression of useful information you point out:

E3S, GAMPO-P (ours), change vs. λΓ=0\lambda_\Gamma=0m=1m=1m=2m=2m=4m=4m=8m=8 (selected)m=16m=16
Regret (1597 at λΓ=0\lambda_\Gamma=0)−0.9−3.5−11.3−15.1−14.2
Final value (0.715 at λΓ=0\lambda_\Gamma=0)−0.016−0.014−0.007−0.001−0.001

Where the Bayes profile would suggest m=4m=4, the measured profile of the trained policy rounds up to m=8m=8 and so keeps the final value that m=4m=4 loses. Since the choice after 25 updates already agrees with the final one (on E3, both 8 and 16 are best windows), a pilot of a quarter of the training budget suffices.

4. Revision. We will add the rule to Section 5, next to the windowed loss, and the validation to Section 7 and Appendix A.5.

Thank you again. Your question helped us turn the window rule into a complete procedure.

Reviewer_sSQFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of analyzing and optimizing memory in LSTM-based agents operating in partially observable environments. The theoretical analysis establishes a connection between memory sensitivity and errors in hidden-state estimation and control.

Critical strengths

Strong theoretical analysis.

Critical weaknesses

Despite its strong theoretical analysis, the paper critically lacks a comprehensive experimental evaluation and a clear connection between the obtained results and practical applications. In the conclusion, the authors mention potential evaluation on POPGym and comparisons with other baselines; however, even this would be insufficient. POPGym is a vector-based environment, and it is necessary to demonstrate the effectiveness of the proposed approach in visual environments as well, such as POPGym Arcade [1] or MIKASA-Robo [2]. The same applies to comparisons with baselines that develop recurrent memory mechanisms [3, 4, 5, 6].

Without a comprehensive experimental evaluation, it is impossible to assess the practical significance of the work.

  1. Wang, Zekang, et al. "Investigating Memory in Model-Free RL with POPGym Arcade." arXiv preprint arXiv:2503.01450 (2025).
  2. Cherepanov, Egor, et al. "Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.
  3. Morad, Steven, et al. "Reinforcement learning with fast and forgetful memory." Advances in Neural Information Processing Systems 36 (2023): 72008-72029.
  4. Le, Hung, et al. "Stable hadamard memory: Revitalizing memory-augmented agents for reinforcement learning." International Conference on Learning Representations. Vol. 2025. 2025.
  5. Cherepanov, Egor, et al. "Recurrent action transformer with memory." International Conference on Learning Representations. Vol. 2026. 2026.
  6. Cherepanov, Egor, Alexey Kovalev, and Aleksandr Panov. "ELMUR: External layer memory with update/rewrite for long-horizon RL problems." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors

No questions.

2 comments

Thank you for pushing us toward visual benchmarks and dedicated memory baselines. We now evaluate on both suggested suites against all four suggested memory models [3–6] and report every run. In the tables, (ours) marks a standard learner trained with our influence loss; all other rows are baselines.

1. POPGym Arcade (pixels, online PPO). Four official tasks with partial 128 × 128 RGB observations, 2202^{20} transitions, three seeds and 128 fresh final-policy episodes per seed. All methods share the upstream CNN, PPO settings, memory wrapper and transition budget; parameter counts differ (LSTM 11.4M, FFM 6.2M, SHM 4.6M), and SHM follows its paper's sampling with memory size 128.

MethodCartPole-ECountRecall-ECountRecall-MAutoEncode-E
LSTM-PPO (baseline)0.889 [0.837, 0.953]0.356 [0.349, 0.367]0.176 [0.125, 0.237]0.286 [0.260, 0.307]
LSTM-PPO + influence (ours)0.841 [0.759, 0.938]0.324 [0.313, 0.342]0.194 [0.169, 0.228]0.260 [0.256, 0.263]
FFM-PPO [3]0.832 [0.739, 0.899]0.307 [0.255, 0.397]0.137 [0.118, 0.154]0.290 [0.274, 0.301]
SHM-PPO [4]0.775 [0.740, 0.823]0.219 [0.202, 0.236]0.085 [0.066, 0.123]0.295 [0.280, 0.313]

Mean return [95% seed-bootstrap interval].

Our regularized LSTM has the highest mean return on CountRecall-M, and both LSTM variants stay ahead of FFM and SHM on three of four tasks. Resetting the actor state before every decision measures how much each policy relies on memory. On CountRecall-M this costs our policy 0.165 [0.154, 0.180], more than LSTM-PPO (0.139), FFM (0.105) or SHM (0.076): the loss keeps the count the task needs. On AutoEncode-E, which requires recalling the whole card sequence, it costs only 0.006 versus 0.099 for LSTM-PPO, so this task calls for a smaller weight.

2. MIKASA-Robo with RATE [5] and ELMUR [6]. RATE and ELMUR are offline sequence models, so all methods are trained in the authors' framework on the released datasets (1,000 demonstrations; two RGB cameras; no joint or oracle state) with ELMUR's released schedule. ELMUR keeps its released architecture; RATE and BC-LSTM use the framework's settings, untuned for MIKASA-Robo. Our variant adds the influence loss to BC-LSTM. Final checkpoints, without selection, are evaluated on 100 fresh episodes per run.

Success (mean ± SEM, 3 seeds)RememberColor3ShellGameTouch
ELMUR [6]0.79 ± 0.030.99 ± 0.00
RATE [5]0.79 ± 0.060.75 ± 0.07
BC-LSTM (baseline)0.19 ± 0.020.70 ± 0.08
BC-LSTM + influence, weight 0.01 (ours)0.23 ± 0.010.75 ± 0.09
BC-LSTM + influence, weight 0.1 (ours)0.18 ± 0.030.23 ± 0.01

Published values are 0.89/0.96 for ELMUR and 0.65/0.92 for RATE. ELMUR's external memory leads clearly. At weight 0.01 our loss matches or exceeds BC-LSTM on both tasks, whereas weight 0.1 is too strong for ShellGameTouch.

3. Practical use: one memory audit for every architecture. Our finite intervention behind Proposition 1 and Theorem 20, replacing only the cue frames and measuring the change in later actions, applies unchanged to recurrent and transformer memories and needs only teacher-forced passes. On RememberColor3, a cue of a different colour changes the later actions of ELMUR and RATE by 0.02–0.10 (mean L2), against at most 0.002 for a same-colour cue: both store the colour, not nuisance detail. The audit also explains the two weights before any rollout. At 0.1, actions become nearly invariant even to the current frames (≤ 0.001): the optimizer closed the observation pathway. On ShellGameTouch at 0.01, the responses to the cue (0.002–0.021) and to the current frames (0.023–0.026) stay at BC-LSTM's level (0.006–0.009; 0.029–0.031), while the response to other inputs six or more steps old falls about 30-fold (0.0001–0.0002 versus 0.004–0.005; ELMUR 0.014–0.016). There the weight tunes a selective memory that keeps decision-relevant cues and suppresses nuisance history, the trade-off Corollary 6 formalizes. The audit also reveals what success rates hide: on RememberColor3, where every LSTM variant stays near 0.2, all regularized seeds and one BC-LSTM seed ignore the frames entirely, whereas ELMUR and RATE respond strongly to the cue. In practice, our analysis gives a model-agnostic check of which history a policy uses, and a single weight to control it.

4. Revisions. We will add both studies, the audit protocol and all per-seed records, mark our variants and the baselines in every table, and state that the visual experiments insert our influence loss into standard learners (PPO, behaviour cloning) to isolate its effect across memory architectures.

Thank you again for your constructive review, which has substantially improved the quality of our paper. As you suggested, the revised paper now evaluates our influence loss on POPGym Arcade and MIKASA-Robo against FFM, SHM, RATE and ELMUR (Table 2 and Appendix A.7). With the author–reviewer discussion period drawing to a close, we would be grateful to know whether our response has addressed your concerns, and we look forward to your final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their careful reading and constructive comments. We are glad that Reviewer sSQF found the theoretical analysis strong and that Reviewer 4QZS recognized its characterization of how history perturbations propagate to belief and control. The reviews raised three shared questions: how the theory justifies the influence loss and which history it should suppress (4QZS, tv8M), whether the guarantees extend beyond filter contraction (tv8M), and what the experiments show about recall, visual environments and dedicated memory architectures (all reviewers). We answer each with new theory and new experiments, now integrated into the revised PDF, where new or revised text is shown in blue. Throughout, GAMPO-P and GAMPO-KF are our methods, "(ours)" marks a standard learner trained with our influence loss, and all other methods are baselines. All new protocols were fixed before training, and every run is reported.

Summary of revisions

  1. From the memory tail to the regularizer (Sections 3–5). The new Proposition 4 shows that the memory tail bounds how much the action law can depend on old inputs: for histories that share their last mm inputs, TV{πh(⋅∣Hh),πh(⋅∣Hh′)}≤DX22Thπ(m)≤DXLℓ,h22Th(m)\mathrm{TV}\lbrace\pi_h(\cdot\mid H_h),\pi_h(\cdot\mid H_h')\rbrace\le\frac{D_X}{2\sqrt2}T^\pi_h(m)\le\frac{D_XL_{\ell,h}}{2\sqrt2}T_h(m). Our influence loss is a sampled surrogate of the policy tail Thπ(m)T^\pi_h(m), and Corollary 8 yields a window rule: suppress only history older than the window the filter retains. The loss in Eq. (9) now carries this window; m=1m=1 recovers the submitted loss.

  2. Guarantees beyond filter contraction (Section 4, Appendix C). Theorem 6 now holds path-wise with step-wise Lipschitz constants κh\kappa_h, covering delayed recall (κh=1\kappa_h=1) and reveals (κh=0\kappa_h=0). Proposition 7 bounds the active-sensing belief error in log-odds, Proposition 12 replaces the Lipschitz policy head by a margin argument, and Proposition 13 bounds recall value by cue transport, Vπ≤12+(12−q)εˉV^\pi\le\frac12+(\frac12-q)\bar\varepsilon.

  3. The window rule on E3 (Section 7, Table 1, Figure 2; Appendix A.5). In 20 new paired seeds, the regret change of GAMPO-P relative to λΓ=0\lambda_\Gamma=0 goes from +187 [165, 209] at m=1m=1 to −11 [−28, 5] at m=16m=16, and the value after an erased terminal observation is preserved (0.748 in both). Where old inputs are pure nuisance, the rule selects m=1m=1: in the prespecified fast-refresh confirmation, the loss lowers regret by 300.8 [272.2, 327.4] (16%). On E4, replacing the full-path Jacobian by the forget-gate proxy raises regret by 503.6 [470.9, 538.0].

  4. What governs long-delay recall (Section 7, Figure 3; Appendix A.6). The cue's full-path transport at initialization, 0.14, 0.034 and 6×10−66\times10^{-6} at delays 8, 16 and 64, predicts where recall is learnable, and Proposition 13 matches the observed value in all 35 new runs. GAMPO-KF, whose consistency loss supervises the belief at every step, learns the 64-step recall in all five new runs (0.874). On these trained policies, the margin condition of Proposition 12 holds on every held-out E1 history, and its bound (0.114) covers the observed value gap (0.026).

  5. Visual benchmarks and memory baselines (Section 7, Table 2; Appendix A.7). On four POPGym Arcade tasks with 128×128128\times128 RGB observations, our regularized LSTM attains the highest mean return on CountRecall-Medium, and both LSTM variants lead FFM and SHM on three of four tasks. On MIKASA-Robo, with all methods trained in the RATE framework alongside RATE and ELMUR, the influence loss at weight 0.01 matches or exceeds BC-LSTM on both tasks. Our cue-swap audit applies unchanged to all five memory architectures; on ShellGameTouch, the calibrated loss keeps the responses to the cue and to the current frames at the BC-LSTM level while reducing the response to other inputs six or more steps old more than 30-fold.

  6. Presentation. Every table marks our methods and the baselines; the complete comparisons and ablations of the submitted Tables 1–2 are in Appendix A.1; FFM, SHM, RATE, ELMUR, POPGym Arcade and MIKASA-Robo are discussed and cited; and a revision summary opens the appendix. Sections 1–6 were tightened to make room for the new material; edits that only shorten existing text are not highlighted.

Numbering. Our individual replies follow the submitted numbering. In the revision, Theorem 5 → Theorem 6, Corollary 6 → Corollary 8, Theorem 20 → Theorem 24 (its extension to cue noise is the new Proposition 13), and Proposition 21 → Proposition 25.

We hope the revision addresses the reviewers' concerns, and we would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.