acceptodds
Under review as a conference paper at ICLR 2027

Train Short, Act Long: Multistability and the Horizon Independence of Recurrent Policies

Abstract

In partially observable reinforcement learning (RL), a decision often depends on a cue seen many steps earlier, and training at such horizons is costly. Ideally, a policy trained on short horizons would remain optimal at arbitrarily long ones, making the training horizon a matter of convenience. We formalize this property as *delay-horizon independence* (DHI): optimality at every horizon, at a memory, computation and precision cost that is bounded in the horizon. We study agents that carry the cue in a recurrent memory of fixed size, whose cost does not grow with the delay. This paper characterizes DHI through the dynamics of that memory, in theory and on benchmarks. Although quantified over all horizons, DHI reduces to a property of a single map: a recurrent policy is DHI if and only if it admits a continuous readout of the memory, invariant under the idle dynamics and separating the cues. Monostable memory then precludes DHI at any parameter values, whereas multistable memory enables it. Monostability covers all contracting affine recurrences, hence state space models and gated linear recurrent neural networks (RNNs). Under arbitrarily small perturbations of the idle dynamics, multistability is the only mechanism supporting DHI at bounded memory. On the T-maze, the effective horizon of monostable models resists hyperparameter search: selected horizons fall short by up to when retrained. Multistable models trained on short horizons stay optimal up to , on the T-maze and on MiniGrid. Current parallelizable RNNs thus trade horizon independence for scalability: parallelizable multistable architectures are the direction this points to.

Primary area:
reinforcement learning
Keywords:
partially observable reinforcement learning, POMDP, long-horizon reinforcement learning, recurrent memory, RNN, state space models, linear recurrent neural networks, multistability, length generalization, credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7dVaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether recurrent policies trained on short delays can still work over much longer ones. The authors show that monostable memory eventually forgets, while multistable memory can retain information indefinitely. This distinction matters because many popular parallelizable RNNs have monostable memory. The authors conducted experiments on T-maze and MiniGrid tasks whose findinds support the central claim of the paper.

Critical strengths
  1. The question of whether a policy trained on short delays still works at much longer ones is clear and practical. The criterion the paper gives for checking this is also simple, since it only depends on whether the memory dynamics are monostable or multistable.
  2. Testing memory all the way up to 10⁶ steps with 30 seeds is convincing. I especially liked Figure 3, which shows that the effective horizon of monostable models varies so much across seeds that hyperparameter search can’t control it.
  3. The observation that popular parallelizable RNNs are monostable by construction is timely and useful for people choosing architectures for memory-heavy RL tasks.
Critical weaknesses
  1. The delayed-cue POMDP setting seems narrow, and the authors introduce it without positioning it against prior work or discussing its relevance to modern deep RL benchmarks. The benchmarks use, including T-maze, MiniGrid-Memory and MiniGrid-GoToDoor, are all simple versions of the same cue-delay-decision structure.
  2. That multiple attractors can store discrete memories is well known in dynamical systems and neuroscience (e.g., Hopfield, 1982; Sussillo and Barak, 2013), so the main claim of this paper appears limited in novelty.
  3. Only a fraction of the trained GRU and BRC models end up multistable, and the paper does not propose a way to make training reliably reach multistability.
Questions for authors
  1. Could the authors discuss how the delayed-cue POMDP setting relates to existing memory benchmarks in deep RL? How common is this cue–delay–decision structure in those tasks?
  2. Do the authors expect their conclusions to hold on more complex memory tasks, such as ones with multiple cues, continuous cues, or memory that needs to be updated over time? Any experiments beyond the T-maze style tasks would help show the broader relevance.
  3. I'm curious how multistable models like BMRU do compared with monostable ones on standard POMDP benchmarks where memory needs to be graded or updated often. Is there a tradeoff there?
  4. Since only some of the trained GRU and BRC models end up multistable, do the authors have ideas for a training procedure that would get standard cells like GRU or LSTM there more reliably?
  5. Could the authors comment on how the framework would extend to tasks where the cue is continuous rather than discrete?
  6. How robust are the multistable models in practice to larger perturbations, like strong observation noise or distractors during the delay? It would be nice to see how performance changes as noise increases, compared with the monostable models.
Reviewer_NShyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper discusses the question of whether a policy trained on short-horizon versions of a task can generalize well when tested on longer-horizon versions of the same task. The paper defines a family of POMDPs, delayed-cue POMDPs, where the agent observes a cue, then a series of uninformative observations. Afterward, the agent needs to take an action based on the earlier cue. The number of steps between when the agent observes the cue and when it has to react is denoted as T. The paper defines a delay-horizon independent (DHI) policy as the one that is optimal for all T>=TminT >= T_{min} and that its memory footprint and computational complexity are independent of T. The paper argues that a recurrent memory with one global attracting fixed point (monostable) cannot be DHI, and that a recurrent memory with two or more attractors (multistable) can be DHI. Finally, the paper provides experimental results on the T-maze environment and two minigrid tasks where the tested horizon was up to 106.

Critical strengths

The delayed-cue problems are common in partially observable RL environments and are interesting to study.

Critical weaknesses

The paper tries to formalize when recurrent architectures can keep a cue in memory over arbitrary long delays. However, the setting being studied in the paper is very simple, and the paper offers minimal new insights even in this simple setting. The paper studies the setting where there is a single cue followed by a constant, uninformative idle phase. The main message is that a memory with one attractor eventually forgets while a memory with multiple attractors does not. This is already well known in the literature on fading memories and multistable RNNs (see 1, and 2). The paper mostly tries to add new formalization around these known results. There is a lot of notation here for a very simplified setting with observations and actions fixed after observing a cue, and it is not clear the benefit this brings. There is additionally some unjustified parts of this formalization, particularly for the extension to the setting where the observations can change after the cue.

The paper tries to slowly outline a set of definitions and theorems. However, there is quite a lot of notation introduced. Not all of those choices were clear. Below outlines some issues.

  1. The notion of a readout function with decision-relevant representations is given. Why does this need to be defined? It is clear that later you need to separate h converging, and how small changes in h might not change the policy. But this specific decision-relevant component was not clear.

  2. Property (b): should the memory and per-step computation be bounded by T or constant regardless of T?

  3. The definitions are asymptotic in T. But for any finite T, even a very large one, we could still have a policy that distinguishes between cues. This should be clarified.

  4. Theorem 3.7 is hard to read but also seems to be saying something quite straightforward, given we have multiple basins. Also, I am not sure k(sigma) is defined. The text below also says this theorem is verified, but I am not sure any of these quantities are measured.

  5. Proposition 3.8 moves to the setting where the observations are not fixed (beyond idle). This is actually important, since the problem setting with fixed observations and actions is quite unrealistic. But the statement itself is imprecise, given the assumptions are inherited from Theorem 3.7 but cannot directly be used. The proof is also short and potentially insufficiently formal, leaving out steps.

  6. The proofs in general were either quite short (and at times too informal) or in other cases were hard to follow (Lemma 3.2)

Overall, this paper is not yet ready and likely needs more work to get insights beyond what is known for monostable and multistable RNNs. It is important to consider the “so what” for this paper, and how it helps the reader improve RNNs for RL.

References: [1] Vecoven N, Ernst D, Drion G (2021) A bio-inspired bistable recurrent cell allows for long-lasting memory.

[2] Lambrechts, G., De Geeter, F., Vecoven, N., Ernst, D., & Drion, G. (2023). Warming up recurrent neural networks to maximise reachable multistability greatly improves learning. Neural Networks

Questions for authors

Clarifying the points mentioned in the weakness above would be appreciated.

Reviewer_on1LOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies tasks where an agent sees a cue and must use it later. During the interval between seeing the cue and using it, the cue is no longer available. The paper defines delay-horizon independence (DHI) as the ability of one recurrent policy to remain optimal no matter how long this interval becomes, without requiring more memory, computation, or increasingly precise numbers. The paper examines which kinds of recurrent models can preserve the cue. If every possible memory eventually approaches the same state, information about different cues will eventually be lost. If the model has several stable memory states and stores different cues in different states, it can preserve the information indefinitely. Under a strong requirement that memory must survive every small disturbance, the paper argues that having multiple stable states is necessary. This suggests that the contracting SSMs and gated linear RNNs studied in the paper cannot remember indefinitely, whereas nonlinear models with multiple stable states can. The experiments test several recurrent architectures on T-maze and MiniGrid tasks. Models with only one stable state eventually forget the cue, although the point at which they fail varies considerably across training runs. Successful models with multiple stable states retain the cue for all tested intervals, up to 10610^6 steps. However, finite experiments cannot demonstrate that they will remember for an unlimited number of steps, and training does not always produce the required stable memory states.

Critical strengths
  1. The main contribution is not merely that a compressed history must retain decision-relevant information. It identifies when one fixed-size recurrent implementation can preserve a cue for every delay without increasing memory, per-step computation, or numerical precision. The monostability obstruction is strong under its constant-input and global-attractor assumptions; the positive result states the needed cue-to-basin encoding and invariant readout conditions; and the robust necessity result clearly depends on a strong perturbation premise. This dynamical and architectural characterization is consequential to my acceptance assessment because it supplies a concrete realizability boundary that general information-state sufficiency results do not provide.

  2. The measured memory regime predicts the observed effective-horizon regime: monostable models eventually fail at finite and highly seed-dependent horizons, while successful multistable models retain performance through (10^6). The comparisons also help separate attractor structure from model capacity and long transients. This correspondence is consequential to my acceptance assessment because it tests the proposed mechanism rather than reporting benchmark scores alone. The evidence remains finite, does not prove infinite-horizon DHI, and does not show that training reliably finds a successful multistable representation.

Critical weaknesses

Definition 2.1 does not require rewards, transition dynamics, discounting, or optimal nonterminal behavior to remain consistent across horizons. A longer-horizon family member can therefore preserve the same cue-independent observations and final cue-to-action rule while adding an intermediate reward or control requirement for which the original policy is suboptimal. Cue retention and final-action correctness then do not imply global optimality, contrary to the sufficiency step in Lemma 3.2 and consequently the conclusions of Theorem 3.7 and Proposition 3.8. This gap is consequential because those results are the formal bridge from recurrent memory dynamics to DHI as global policy optimality. A constructive repair is to define an explicit horizon-extension operation under which increasing (T) only inserts cue-uninformative, reward-irrelevant evolution while preserving rewards, transitions, discounting, and all other optimal behavior. Alternatively, the authors could weaken the claimed property to correctness at the delayed decision.

Questions for authors
  1. Is the intended family subject to a horizon-extension consistency condition that is absent from Definition 2.1? Please give the exact added assumption and revise the affected statements and proofs. In particular, explain whether each longer-horizon POMDP is obtained only by inserting cue-uninformative, reward-irrelevant steps while preserving transitions, rewards, discounting, and nonterminal optimal behavior, or whether DHI will instead be weakened to delayed-decision correctness.

  2. How do DHI and the invariant readout relate to exact and approximate information states—especially Subramanian et al. (JMLR 2022; https://www.jmlr.org/papers/v23/20-1165.html) and Patil et al. (AISTATS 2024; https://proceedings.mlr.press/v238/patil24b.html), which appear particularly relevant here? Briefly situating the present contribution relative to these works would provide a more complete account of the pertinent literature. Please state whether the recurrent state, the readout, or an augmented state satisfies reward and next-state sufficiency, and identify precisely what is new beyond the established sufficiency principle. My assessment depends on whether the novelty is correctly limited to horizon-uniform bounded-precision realizability, attractor structure, and architecture classification

  3. What is the precise scope of the variable-input and architecture claims? Do the MiniGrid environments and trained policies satisfy the separated, forward-invariant-region and invariant-readout premises of Proposition 3.8, or are those experiments only finite on-policy evidence consistent with the proposition? Likewise, is the SSM exclusion limited to the analyzed contracting, state-affine recurrences under constant idle input, rather than all SSMs, parameter regimes, and varying-input realizations? The answer determines how broadly the theory explains the empirical results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.