- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Minimal Recurrent Behavioral Memory (MRBM), which takes a theoretical approach to questioning what is needed for memory and context in robot learning tasks. It proposes a series of proofs that seeks to find the lower bound to the information that is needed for a given task, under certain assumptions. At heart, the paper decomposes memory into two components, in my mind loosely equivalent to short term and long term memory: the current class stores memory required by the immediate action, and the behavioral memory stores information that should persist further into the future.
Critical strengths
This paper is well motivated, and I really like it's problem formulation. I takes a very principled decomposition to what is the minimum of memory we need to store to maximally perform in a task that requires such memory. In other words, minimize storage (bitwise information) while maximizing sufficiency (to solve the task). I believe this alone sets forth a clear framework for how to think about both memory and context in learning.
The idea of incorporating future observations into consideration for what should be learned is a strong point -- this makes a lot of sense intuitively, as some memory information may only become clear once we see the full task through and is appropriately highlighted as an important point throughout.
Critical weaknesses
While the paper presents a nice theoretical formulation, it's applications are unclear. The writing is filled with empirical numbers and readouts that seem rather specific to the task at hand (ln364, ln349-350, ln320-321 for example). Indeed, it is impossible to match theory with practice always precisely, but in such cases, I am not fully convinced by the experiments that a) the set of assumptions used for the proof is actually necessary in practice, and b) more broadly, that the proposed method has practical applications to real-world robotics.
Put differently, I think the paper's framework and positioning is strong; however, the design decisions and instantiations make me question it's applicability and relevance. I feel that the whole paper is motivated from a simple robotics example, but the method is extremely overfit to solving that example in particular. It is not clear to me as the reader how this method would extend to tasks that have more information requirements, for example beyond 2 bits. The experimental evidence does not demonstrate the broad applicability nor robustness needed to convince me that this framework is viable and practical for applications of memory in robotic tasks. For longer horizon tasks and real-world image observations, it is unclear to me how such a method would cleanly extract out the bits of information to preserve, propagate, and match. The appendix has a thorough number of proofs, though they do not contribute to me being convinced of the general practicality of this method.
Overall, the lack of comparison and connection to other baselines on real-world robotics tasks makes me question whether this framework sees any practical benefit over existing memory architectures. I strongly encourage the authors to perform more real-robot experiments, on harder, long-horizon tasks, and compare against more learning-based methods such as [1] and prove practical and empirical gain. If this is infeasible, I at least encourage authors to do more thorough investigation/comparison to improve the practicality of the proposed framework.
[1] Gated Memory Policy, https://arxiv.org/abs/2604.18933
Questions for authors
- How does this method extend to real-world applications, where information bits possibly extend beyond 2->1->0? Can the authors please detail out a realistic setting in which their framework, or a learning-based approximation in high-dimensional space, would be practically usable without needing to manually define constraints/information? Which parts of the framework would scale, and which would be less scalable?
- I am concerned that the theoretical establishments and assumptions limit the broad applicability and practically of the method. Specifically, if the authors want this to work in robotics, can the authors please walk through a concrete example of how their method would apply in a long-horizon task? What is a task where the assumptions hold, and what is a task where the assumption would break? Which components are more brittle than others?
- As an overall note, I would suggest the authors not dive so deeply into jargonic terms so early, in both appendix, and especially in the introduction. It is extremely unclear what a lot of phrases mean in the introduction, thus hindering the understanding of the paper. For example, please see ln 46-48 in the introduction. Further, in the teaser Figure 1, "recurrent target that accounts for future observations" is a significant part of the design, yet this phrase is hard to parse. Rate level meanings and significance should also be explained earlier, if the introduction is chosen to start so technical.
- Relatedly, can the authors please clarify what the main takeaway from each table and figure is? It's uncelar which is the baseline, and what the numbers mean relative to performance. It's difficult to parse what is actually helping.
- I also think figures and tasks would benefit from visual grounding -- could the authors please explain more why the task they chose is appropriate for what they want to demonstrate?
4 comments
Hi Authors, I appreciate your careful response and clarifications to my question. To be concise -- I am still not convinced my core concern on practicality is addressed.
Taking the authors' response specifically in response to W1/Q1/Q2 (2nd response): "Section 7 now gives the workflow: specify the expert and task model;". If my understanding is correct, this doesn't feel too scalable. In a real-world visuomotor task, defining the relevant behavioral distinctions, what information is available in the observations, and when that information will become available again seems to require substantial task-specific knowledge, is quite the undertaking.
This abstracts away a lot of the decision making and difficulty of the problem, and primes the robot in terms of what it needs to remember. It remains unclear to me how one would construct such a model for a more complex task, a long horizon task, or even a language-specified task (e.g. open-vocabulary).
I understand that enumeration is confined to offline certification, rather than learning or deployment. However, my concern is broader than the computational cost of enumeration. It is whether the model needed for certification can be constructed faithfully in realistic settings, and whether the resulting certificate translates into a useful decision about the learned policy. For example, the three-bit experiments show that a theoretically sufficient codebook does not necessarily support reliable learning. Thus, while the requirement provides a lower bound on capacity, I am not yet convinced that it provides actionable guidance for choosing a memory architecture or capacity in practice.
Nonetheless, I find the theoretical formulation interesting. However, the clarifications mainly explain how to apply the framework under a specified task model, but my concern is upstream -- the specification of the task model seems like a hard assumption that undermines this method's practicality, especially as diagnostic. I don't believe my concerns were sufficiently resolved, and will maintain my recommendation.
Reply to the follow-up (1/2)
Thank you for the follow-up. New Appendices E.10–E.12 address how a task model can be constructed from recordings, how that construction affects policy diagnosis, and how visual history measurements can guide a design choice. Analysis plans were recorded in advance and procedures frozen before confirmation; the supplement documents development changes and reuse of recordings.
1. A finite model built without simulator labels. For the ten low-dimensional recordings of Appendix B.2 we rebuilt the task model from measured observations and expert commands, retaining the recorded cue and phase channels. Behavioral distinctions are clusters of the commanded waypoint and observation classes are clusters of the measured end-effector position, fitted on half of the episodes and frozen; the solver is unchanged. Three recordings served for development and the other seven were run once.
Requirement. Of 261 steps, 253 give exactly the requirement of the labelled model, and across the 70 waiting steps the largest difference is 0.002 bit. Three approach steps in the two tasks with four grasp classes differ by two bits.
When a class becomes observable again is inferred from the recorded observations under the declared tolerance. In readout-3, the labelled model requires one bit at steps 11–13; the reconstruction requires one bit at step 12 and three bits at steps 11 and 13.
The recovered distinctions. At 226 of the 261 steps, observation, behavior and memory classes coincide. The differences arise from unresolved arm sag in Task A (28 steps, below the 5 cm tolerance), a one-step shift in grasp timing in the four-grasp-class tasks (6 steps), and the expert's random hover height becoming visible (one step).
Diagnoses of trained policies. Without retraining, we re-scored all 1,032 eligible saved policies, excluding eight engineering trials by a pre-specified rule. Under the main-text sufficiency gate, 1,029 receive the same verdict (475 pass both, 554 fail both). The three changes occur in the 16-class task: of four runs passing under the labelled model, one passes when the reconstruction tests grasp one step later. Task A verdicts remain unchanged, although second-wait code rates rise by up to 1.0 bit when sag is not counted as observed. Appendix E.10 reports the per-task statistics and held-out variant.
Scope and sensitivity. The cue and phase channels are instrumented inputs and episodes are time-aligned. Results change when the tolerance moves from 5 cm to 2 or 10 cm. At 512 hidden masses, the fitted model covers only 5% of held-out sequences, limiting that dataset's independent validation. On the development recording of readout-2, 11 of 40 verdicts change for the same one-step reason.
2. Visual demonstrations without phase, delay or color labels. On RememberColor with three colors, a frozen procedure receives only camera frames, end-effector position and expert commands. It locates the first step at which commands differ and measures how much earlier frames improve held-out prediction there. Developed on one recording, it was run once on two recordings with new scenes and delays it was not given (20 and 12 steps), locating the first differing step at 26 and 18. Squared error fell from 0.099 to 0.015 and from 0.106 to 0.019: reductions of 0.084 [0.079, 0.088] and 0.087 [0.083, 0.092], and 0.077 on both recordings across unseen arm poses. Gains concentrate in the first five cue frames; later single frames give −0.001 to 0.000. With histories exchanged between episodes, gains are at most 0.0005 in magnitude.
Reply to the follow-up (2/2)
A design choice made before training. A fixed rule applied only to training episodes of the 12-step recording selected 14 frames, reaching the latest frame with a sufficiently large positive prediction gain (Appendix E.11). The same network was trained with 7, 14 and 18 visible frames, two seeds each. Held-out squared command error at the first differing step is 0.132, 0.017 and 0.016. Seven and 14 frames differ by 0.115 [0.109, 0.122], and 14 and 18 by 0.0004 [−0.0008, 0.0016]. Intervals resample held-out episodes after averaging the two seeds. This supports the selected window in an offline comparison. Intermediate windows, closed-loop success and computational savings were not measured.
3. A lower bound from the observation–action law. The best of three pixel discretizations covered 64% of held-out histories at the first differing step; none yielded a reliable finite model. Proposition 6 (Appendix E.12) gives a lower bound without constructing a finite partition. It sums the earlier history's additional predictive information at each future decision, conditioning only on observations and actions available by that decision. It applies to any exact recurrent realization in Definition 2.
Under the ten low-dimensional benchmark models' enumerated joint distributions, this bound equals the certificate at every step: 261 confirmation and 104 development steps. On pixels, a separate measurement gives held-out log losses of about 0.14 bit with cue frames and 1.58 bits without them when predicting clustered action labels. The 1.44-bit improvement measures the earlier history's contribution for the fitted predictors; it is not a certified memory bound. The five-color losses are 0.61 and 2.32 bits.
4. Scope. The low-dimensional experiments connect data-built models to the paper's requirement and policy diagnosis, including changed verdicts. The visual experiments recover behavior-relevant timing and guide a history-window choice before training. Their evidence covers one benchmark with one cue and one decision; automatic construction for long-horizon or open-vocabulary tasks remains untested.
5. Testing the model against recordings. The memory diagnostic exposed a one-step pixel-capture offset that correct placement alone missed (Appendix E.1): the model required one bit through a wait, while every seed carried zero bits and still placed correctly. This prompted a direct observation probe and correction of the recording timing.
6. What the certificate adds once a model exists. The requirement and the sufficiency check distinguish capacity, content and memory paths.
- Capacity. A three-bit requirement rules out K < 8 for exact reproduction. At K = 8, 10, 12, 16 and 24, the numbers passing our empirical gate out of 24 are 0, 1, 5, 15 and 23; passing runs carry 2.99–3.14 bits. The requirement sets the feasible floor, and the sweep measures how capacity above that floor affects learning reliability under the tested configuration.
- Content. On RC9 the code carries 3.34 bits against a requirement of 3.17 but retains 2.23 bits of color. Exceeding the rate floor does not ensure the required content.
- Location. A code at 1.06 bits under a two-bit requirement, with high task success, motivates checking memory outside the measured code; success alone does not locate it.
These measurements distinguish cases that task success alone cannot separate. The certificate is not given to the policy. Public-benchmark learners use task-informed supervision (Appendices E.6–E.7). In Section 5, the frozen event-agnostic configuration passes on 36/40 seeds; neither its base objective nor its auxiliary supervision uses latent labels or event annotations.
Thank you for asking for clearer practical uses, evidence beyond two bits and our own tasks, and a comparison with explicit-history policies. The revision adds public pixel benchmarks and real-robot recordings, and explains the results through three questions: what must be remembered, what was learned, and how much excess code rate remains?
W2 / Q5 — Beyond two bits and our own tasks.
On unmodified MIKASA-Robo RememberColor3/5/9, the finite task models require 1.585, 2.322 and 3.170 bits through a five-step delay. A scripted simulator expert supplies demonstrations. Recurrent learners receive images and proprioception, without previous-action inputs, and use task-informed future-action supervision and image translation augmentation (Section 4.2; Appendix E.6).
Passing means meeting the empirical diagnostic, not exact zero error. Information and sufficiency use all recorded episodes; errors use the held-out 20%. Appendix E.6 reports all seed counts, the rate sweep and schedule comparisons. The original three-bit readout result also remains: 23/24 runs pass at K = 24, with rates of 2.99–3.14 bits.
W1 / Q1 / Q2 — Practical use, longer horizons, and components.
The requirement provides an offline reference for choosing memory capacity and distinguishing missing task information from excess rate. Section 7 now gives the workflow: specify the expert and task model; certify what must persist and for how long; measure the learned memory; then evaluate control in closed loop. Enumeration is confined to certification. Learning and deployment do not enumerate histories; exact computation requires an enumerable model or a separate bound construction. Section 5 measures how longer delays reduce learning reliability at fixed requirements.
Appendix B.11 now accounts for all studies' metered compute, approximately 1,272 GPU-hours. Table 3 states the assumptions. Appendix E.9 connects the component controls to their consequences: a persistent hidden state can bypass the measured code; auxiliary prediction can retain surplus; and the smallest theoretically adequate codebook may train unreliably. These results separate the memory requirement from learning reliability and control performance.
W3 — GMP and community real-robot data.
On the public place-back recordings, an eight-direction model requires three bits for median intervals of 8.0 and 12.9 seconds. These include 5.5 and 10.6 seconds with no instantaneous demand. Finer position partitions change the level while preserving the retention profile in the tested models (Appendix E.7).
The released GMP gate opens in 82% of carry frames and 4–7% of frames in the middle of the saucer phase, although both phases require retaining position. GMP keeps history when the gate closes: the gate measures access, while the certificate measures information that must remain available. This explains their complementary roles without interpreting a closed gate as lost memory (Section 6). Appendix E.8 likewise includes the history channel when assessing explicit-history policies; its local-attention experiments do not identify a minimum window.
Small learners retain 1.2–1.4 of the three required bits on these recordings. We have not evaluated hardware closed-loop performance or jointly trained GMP with the certificate.
Q3 / Q4 — Presentation and figure takeaways.
The abstract and introduction now start with a robot remembering and later forgetting a cue. Tasks are explained at first mention, and captions state the findings and compared quantities. Formal terms follow the motivating example. The general response lists all revision locations.