- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to quantify and regularize memory in recurrent policies for partially observable reinforcement learning. It uses the hidden-state LSTM Jacobian to bound historical input influence, connects these bounds to belief and control errors under additional assumptions, and proposes GAMPO, which combines policy mirror descent with sensitivity and information-state losses. In the experiment, they evaluate GAMPO on four synthetic POMDP tasks: delayed recall, conditional recall, a filter-stable chain, and active sensing. Also, using baseline comparisons, regularization ablations, and diagnostic experiments on memory sensitivity and representation quality.
Critical strengths
The paper makes an interesting distinction between being sensitive to past inputs and retaining information useful for decisions. By considering both the cell and hidden states, the analysis captures effects that forget gates alone miss. This helps explain why reducing sensitivity to past inputs can remove distractions or erase important clues.
Critical weaknesses
The theoretical assumptions seems strong and limit where the results apply. The main connection between memory and belief accuracy requires filter contraction, along with accurate decoding and Lipschitz conditions. If I understand correctly, the contraction assumption does not hold in the paper’s delayed-recall and active-sensing tasks. Although some results apply more broadly, this central part of the theory does not cover several important tasks. Also, the theory does not show that the proposed training method meets these assumptions.
In experiment, as authors acknowledged, GAMPO-P performs worse than simpler baselines on other tasks, and its benefit occurs when past information is unnecessary, so it does not demonstrate improved retention of useful memory.
Questions for authors
Can the theoretical assumptions be weakened? In particular, can you obtain useful bounds for delayed recall and active sensing, where uniform filter contraction does not hold?
What causes the poor recall performance of GAMPO-P?
2 comments
I thank the authors for their response. Although the response partially addresses my concerns, I am still not convinced that the contribution meets the bar for acceptance. Accordingly, I will keep my score (including confidence).
Thank you for these precise questions. Both assumptions can be weakened to cover delayed recall and active sensing, and new experiments identify what limits recall. GAMPO-P and GAMPO-KF are our methods; the other encoders of Table 1 are baselines.
1. Path-wise stability instead of uniform contraction. Let eh=TV(b^h,bh) with the one-step defects δh of Eq. (3). If along the realized history the Bayes update is κh-Lipschitz in TV on a domain containing the compared beliefs, the recursion of Theorem 5 gives
eh≤min{1,e1∏j=2hκj+∑s=2hδs∏j=s+1hκj}.
Theorem 5 is the case κh≤ρB<1. In delayed recall every post-cue update is the identity (κh=1), so eH≤min{1,e1+∑s≥2δs}; in E2 the short-cue reveal (C=0) has κ=0 and resets the error. Accurate cue decoding and small one-step defects thus suffice, with no forgetting required.
2. Active sensing in log-odds. In E4 a noisy probe can expand TV (by up to (1−q)/q), so we work with ℓh=logitbh=ℓh−1+λh, where λh=0 after a skip and ±logq1−q after a probe. For interior decoded beliefs, set rh=ℓ^h−ℓ^h−1−λh and Dh=∣ℓ^1−ℓ1∣+∑s=2h∣rs∣; telescoping and ∣σ(u)−σ(v)∣≤tanh(∣u−v∣/4) give
eh≤tanh(Dh/4)≤Dh/4,
exact at zero defect and free of contraction.
3. Control without a Lipschitz policy head. Each Qh⋆(⋅,a) is a supremum of linear functionals of the belief with range [0,Bh], Bh the maximal remaining return, hence Bh-Lipschitz in TV. If the action is ϵh-greedy for Qh⋆(b^h,⋅), two belief substitutions and the performance-difference identity give
V⋆−Vπ^≤∑h=1HEdhπ^[2Bheh+ϵh].
For the binary terminal decisions of E1–E3 a margin argument is sharper: the greedy label is Bayes-optimal whenever eH<∣bH−21∣, so V⋆−Vπ^≤E[∣2bH−1∣1{eH≥∣bH−21∣}]. In E1, ∣bH−21∣=0.4: any decoded belief within 0.4 of the truth recalls optimally. In E4, Bh≤1.05 and Section 2 supplies eh. Both are checkable on trained models: on five retrained GAMPO-KF policies for E1, the decoded terminal belief stays within 0.063 of the exact posterior on all 10,000 held-out histories per seed, so the margin holds everywhere; the path-wise bound e1+∑sδs averages 0.094, and Emin{1,2eH+ϵH}=0.114 bounds the observed gap of 0.026.
4. What limits recall. The influence penalty is not the cause: omitting it changes GAMPO-P's E1/E2 regret by 0.5 [−2.7, 3.5] and −4.2 [−26.9, 15.5], every encoder baseline also stays at chance, and GAMPO-P learns recall at delay 8 (0.876; optimum 0.900). In E1 the only cue-dependent signal is the terminal reward (Proposition 21), which must cross 64 transitions. A new initialization audit (20 seeds) measures this transport: the cue's ∥∂ZH/∂X1∥ falls from 0.14 at delay 8 to 0.034 at 16 and 6×10−6 at 64, and the shared critic's cue sensitivity at delay 64 is 5×10−9. Theorem 20, extended to cue noise q, gives Vπ≤21+(21−q)εˉ for the paired cue-flip change εˉ of the terminal action, tight in our runs (εˉ<10−7, value 0.500 for GAMPO-P). Accordingly, two diagnostic controls (exact-belief critic, chrono-initialized gates) let the cue through in only 2 of 20 runs, whereas GAMPO-KF, whose consistency loss supervises the belief at every step, learns the 64-step recall in all 5 (εˉ=0.94, value 0.874). On E2, an exact-belief critic lifts the four-step branch from 0.51 to 0.65–0.78 while the 64-step branch stays at chance, separating a critic bottleneck from the transport one.
5. The loss and useful memory. On E3, where 75% of observations are erased and history is needed, we restricted GAMPO-P's influence loss to inputs older than a window m, as Corollary 6 prescribes (20 paired seeds). With m=16, regret is 959 versus 970 without the loss and the value after an erased terminal observation is 0.748 in both, against 0.663 for the all-history loss; with m=8, inputs older than the window lose 2–4-fold of their influence while the last seven keep theirs. With the prespecified fast-refresh confirmation (regret lowered by 300.8 [272.2, 327.4]), these results show that the loss removes stale history without erasing decision-relevant memory once its window follows the theory.
Revisions. We will add the path-wise, log-odds and margin bounds to Section 4 (proofs in Appendix C.4), and the recall diagnosis and windowed-loss experiments to Section 7, marking ours versus baselines in every table.