acceptodds
Under review as a conference paper at ICLR 2027

Designing Associative Working Memory for Partially Observable Reinforcement Learning

Abstract

Partially observable RL creates two complementary memory demands: integrating a trajectory into a control state and preserving specific events for later content-based decisions. We study how the latter demand shapes a working-memory interface. Our task-to-memory formulation derives that interface from the relation a later decision must recover, the retrieval cue, and the event completing the content. For transition binding, this process yields Associative Transition Memory (ATM), a slow–fast architecture coupling a recurrent controller to a return-trained associative matrix. ATM treats each completed transition as a memory token: controller context and action form its address, while the resulting observation, outcome, and termination form its content. Two controlled delayed-association tasks test transition-aligned routing efficiency and return-trained selective writing and recall. Explicit transition routing reaches a fixed competence criterion sooner in PPO-update count than a parameter-matched implicit interface. At the matched-budget endpoint, ATM achieves higher five-seed rolling-20 training return under both association loads. With four independent associations per episode, it improves over tuned Transition-FWP by 36.2% ( vs. ) and leads on all five paired seeds. Separately trained no-read and no-write controls remain near the memoryless reference; phase traces show selective updates. These advantages support the task-to-memory view that RL associative-memory architectures should be specified from task-specific information requirements along five dimensions: memory role, address, content, update, and the learning-and-validation contract.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, partially observable Markov decision processes, associative memory, working memory, fast weights

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2JAMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this work, the authors present a model termed ATM to solve POMDPs (RL without full observability). ATM appears similar to standard linear transformers, but replaces the fast parallel linear update with a slow sequential GRU update (e.g., hidden state ht=ktvt⊤h_t = k_t v_t^\top becomes ht=GRU(xt,ht−1)h_t = \mathrm{GRU}(x_t, h_{t-1})). ATM also bolts key/query and delta update operators on top of the GRU recurrent state (e.g., yt=qt⊤hty_t = q_t^\top h_t).

The authors evaluate ATM on two synthetic RL tasks against a GRU and a "Transition-FWP" that is never defined. Then, the authors perform ablations of various components of ATM.

Critical strengths

Learning how to store and recall information via RL objectives is an interesting topic.

Critical weaknesses
  • ATM is a reimplementation of linear attention, except significantly less efficient. In particular, the key-value delta correction appears to be nearly identical to the gated delta net referenced in the related work.
  • The experiments are completely insufficient for ICLR, evaluating two toy environments proposed by the author. Furthermore, the GRU has much greater accuracy than the proposed method (Table 3) on their tasks, yet the authors bold their method as if it is the state of the art method. I am not sure if this is negligence or subterfuge, but both deserve an ethics review.
  • The work is heavily AI-written and filled with nonsense sentences, which contain many technical-sounding but undefined or vague terms. The result is many passages that are difficult to read but simultaneously provide no information to a reader knowledgeable in RL/ML. Consider, for example, this passage from the abstract:

Explicit transition routing reaches a fixed competence criterion 1.95x sooner in PPO-update count than a parameter-matched implicit interface. At the matched-budget endpoint, ATM achieves higher five-seed rolling-20 training return under both association loads. - What is "transition routing", what is "explicit transition routing"? - What is a "fixed competence criteria"? - What is a "parameter-matched implicit interface"? - What is a "matched-budget endpoint"? - What is "five-seed rolling-20 training return"? - What are "association loads"?

  • Combining memory with a PPO objective is written as if it is novel (section 3.3) when it is most certainly not.
Questions for authors
  • Can you explain why you falsely highlight your proposed method in Table 3 when the GRU accuracy is significantly higher?
    • Why would one choose your method (that uses a GRU internally) instead of the GRU itself?
  • Please rewrite the paper, ensuring you define all terms such that a reader familiar with the Sutton and Barto textbook can follow along.
  • For what reason would one choose ATM instead of a gated delta net?
    • Can you compare the return? How about the wall-clock time efficiency?
Reviewer_mHsnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Associative Transition Memory (ATM), combining GRU with a delta-rule associative memory. ATM uses different parts of a completed transition to construct memory addresses and content, updating the memory online to support later decisions. On two controlled delayed-association tasks, the authors report faster learning with explicit key/value input routing and gains over Transition-FWP in some of the evaluated settings.

Critical strengths
  1. The method is partly clear.
Critical weaknesses

Presentation and clarity:

  1. Table 1 overlaps substantially with Figure 1. It adds little explanation of why these design strategies were made, what alternatives exist, or under what conditions they are appropriate.
  2. The observation encoder and value MLP architecture need to be fully explained.

Method and evaluation:

  1. Why set both ηmax⁡\eta_{\max} and αmax⁡\alpha_{\max} to a small const 0.03? How performance and memory retention change when these two forget/write gate are varied separately?

  2. In Eq. (13): Why use MLP computing the value, rather than linear projection as in key and query? In Eq. (9): Why uses LayerNorm rather than L2 norm for the memory readout?

  3. The routing ablation only compares the proposed split with feeding the full transition to both key and value; alternative splits or component-wise ablations would help justify the specific address/content design.

  4. Direct experimental comparisons for FWM, SRWM, and RFWPs are totally missing. The current results mainly support an interface improvement on the tested tasks, but what inprovements within existing RL fast-memory methods are unclear. Therefore, the framework seems incremental.

Questions for authors

see above

Reviewer_oPH3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores the use of an external fast-memory matrix for POMDPs. In particular, their framework explores the use of information from the outcome of a completed transition as a useful inductive bias. When combined with an existing GRU equipped policy, their method is shown to improve performance on two synthetic environments, demonstrating that ATM can leverage effectively store useful prior information within a trajectory for future decisions.

Critical strengths

None

Critical weaknesses
  • The paper is difficult to read and does not properly contextualize itself within the greater context of the POMDP literature effectively. In general, the paper introduces and uses several non-standard terms for standard RL/POMDP mechanisms without properly defining either, making it very difficult to follow along. In the introduction alone, terms such as "Content-addressable", "event binding", "working-memory design", "information contract", "transition binding", "interface prior", "write mechanism", "transition interface", "parameter-matched routing control", "return-trained selective writing", "no-read controls", "no-write control", "phase-conditioned traces", "binding events", "post-transition outcome content", "error-correcting delta rule", "routing sample efficiency", "parameter-matched control", "five-seed path controls", "return-trained selective writing", "task-level query-only diagnostic", "write-phase inputs" are consistently used throughout the paper, and puts an unnecessary amount of overhead on the reader to parse and infer what each of these mean exactly in the context of a POMDP. This issue is further exacerbated by the fact that the authors never properly introduce or formalize any standard RL or POMDP vocabulary or setup, which would have made the paper more digestible for the familiar reader.

  • The results and conclusions are only weakly verified on two synthetic benchmarks the authors created against a weak GRU baseline. With the lack of theoretical support, the contributions of the ATM framework are not strongly supported. There are also several design choices, which the authors assert are crucial to the ATM's performance (surrounding the "five dimensions") that are not properly intuitively, theoretically nor empirically motivated, for example with rigorous ablations.

Questions for authors
  • I am not convinced the core premise of the paper, that POMDPs require two separate complementary memory demands is properly motivated. With enough capacity, why wouldn't a recurrent state representation be sufficient to recall specific prior events for optimal control in the future? What about transformer-based architectures that do not recurrently compress arbitrarily long histories? If the reason is due to compute or memory efficiency, since recurrent methods are limited by their capacities and attention-based methods grow in complexity as the horizon grows, the paper did not make this shortcoming clear, nor do the experiments reflect it.

  • Similarly, it would be nice to include results, even if only on the synthetic environments, on stronger baselines such as using an attention-based architecture for history representation, or larger capacity recurrent networks to verify these points.

  • The synthetic environments CAOR and CAR are underspecified and lack reproducibility. They are also quite abstract in concept and it would benefit the paper to contextualize them with concrete examples to better understand their relevancy. For example, the CAOR environment explicitly hides information from the GRU policy through a separate ztz_t channel that the memory matrix has access to. Is this a realistic setup? What practical RL environment would mirror this setup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.