- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Divide-and-Remember (D&R), a learned visual-memory selection method for history-dependent vision–language–action policies. It motivates memory learning through the conditional mutual information (I(a_t;m_t\mid o_t)), which measures the action-relevant information retained beyond the current observation. D&R recursively applies a shared selector that reduces up to (2K) candidate tokens to (K), preserving selected visual features instead of replacing them with newly synthesized latent summaries. The selector is trained jointly with a π₀.₅ policy using its flow-matching action loss.
On RoboMME, D&R reports 38.58% average success with 64 memory tokens, compared with 32.22% for the adapted HAMLET baseline and 27.93% for FrameSamp. Additional experiments vary the candidate pool and memory budget. Four real-robot tasks yield 35/40 successful trials, compared with 22/40 for FrameSamp.
Critical strengths
- Learning discrete visual-memory selection is a useful contribution with a clear practical motivation.
The method occupies a meaningful position between fixed frame-selection heuristics and learned latent compression. It learns selection from the control objective while preserving the values of selected visual features. Sharing a small selector across a reduction tree is a coherent way to accommodate varying candidate-set sizes without expanding the parameter count.
- The reported gains are substantial in a relevant evaluation setting and are supported by real-robot evidence.
RoboMME directly tests history dependence, making it well suited to the paper’s central question. Under the reported 64-token setting, D&R improves average success over HAMLET by 6.36 percentage points and over FrameSamp by 10.65 points. The improvement is especially pronounced on Imitation: 37.83% versus 19.50% and 22.06%, respectively. The manuscript also acknowledges cases where latent memory remains competitive, which helps delimit the contribution. The real-robot results provide useful additional evidence: D&R succeeds in 35/40 trials while using 1,024 output memory tokens, whereas FrameSamp succeeds in 22/40 with 2,048 tokens.
Critical weaknesses
- The experiments do not isolate the recursive design, and the implemented history pipeline does not support the claim of full-history memory.
Sections 5.1–5.2 formulate selection over the full history and emphasize an unbounded-history construction with consistent offline and online behavior. However, Section 6.3 reveals that the evaluated selector receives a uniformly sampled candidate pool: the main configuration contains 32 frames with 16 pooled tokens each. Consequently, an event excluded by this initial sampling cannot be recovered by the learned selector. This leaves unresolved whether the deployed system maintains a persistent recursive memory or repeatedly selects from an archive of sampled historical frames. These alternatives have different storage requirements, update costs, and failure modes.
The ablations vary candidate-pool size, output budget, token types, and auxiliary loss, but do not compare the recursive selector with a flat learned selector, or with uniform/random patch selection from the identical candidate pool. At (K=64), FrameSamp retains four complete pooled frames, while D&R can distribute its output across 32 candidate frames. Thus, the existing comparison does not determine how much improvement comes from learned selection, broader temporal coverage, or the recursive organization.
Furthermore, the real-robot configuration uses (|H|=2048) and (K=1024). Under Algorithm 1, this requires only one (2K)-to-(K) selection, so these experiments support learned selection but do not validate hierarchical recursion. A matched selector ablation and an explicit online update/storage protocol are important because recursion and long-history scalability are central contributions, not peripheral implementation choices.
-
Inconsistent reporting metrics. The stated evaluation protocol is 50 episodes per task, averaged over three checkpoints and three seeds. Under that protocol, an equally weighted per-task mean must occur in increments of () percentage points. Several entries do not follow this rule, including D&R’s 7.67% on InsertPeg in Table 9 and D&R+pred’s 41.25% on BinFill in Table 6. This mean and standard deviation cannot describe three or nine nonnegative run-level success rates as stated. Table captions also alternate between variation “over seeds” and “over evaluation runs.” These may be transcription errors or undocumented protocol exceptions, but they require reconciliation with per-run success counts.
-
Insufficiently specified baseline adaptations HAMLET’s original method includes time-contrastive initialization and specific moment-token/history settings, whose treatment in this implementation is not documented. The original RB-VLA uses action/proprioception-conditioned belief estimation and largely episodic VLM querying, whereas this paper excludes proprioception and imposes a common π₀.₅ integration. A controlled common-backbone comparison is reasonable, but its results characterize those adapted implementations. They do not, without further evidence, establish that latent memory inherently loses the relevant spatial detail or that RB-VLA generally requires the reported computational cost. HAMLET, RB-VLA.
Questions for authors
Q1 Implementation clarification. At deployment, is the candidate pool reconstructed by sampling an archive of historical frames, or maintained incrementally through persistent tree nodes? Please specify details retained between policy calls, the frame-resampling rule, and how storage and update latency scale with episode length. Plus, how does the evaluated implementation realize the offline–online equivalence claimed in Section 5.1?
Q2 Evidence needed to validate the central contribution. With the same 512 candidate tokens, 64 output tokens, positional information, and policy-training recipe, how does D&R compare with uniform/random patch selection and a flat learned top-(K) selector? These controls would distinguish temporal coverage and learned selection from the additional value of recursion.
Q3 Reporting clarification. Please provide per-checkpoint, per-seed success counts and denominators for Tables 6–9, explain how standard deviations are computed, and reconcile the entries identified in W2. Were any variants evaluated using different episode counts or fewer runs?
Q4 Baseline clarification. Was HAMLET’s time-contrastive initialization retained, and what were its history length and moment-token allocation? Which components and objectives of RB-VLA were preserved? Please distinguish changes required for π₀.₅ integration from optional simplifications, and clarify whether the efficiency comparison concerns the original method or this adaptation.
5 comments
Response to Reviewer fS2a (2/3)
R2 (continued).
(2) Fixed tree (the original main results). As the history grows, we sub-sample it into a pool of fixed size, so that we only maintain a tree of fixed size with a constant number of leaves. Specifically, a fixed number of frames is sampled uniformly from the history observed so far, which gives a candidate pool of fixed size . The pool is re-drawn at every step, so the spacing between its frames widens as the history grows, while the cost of the selector stays independent of the history length.
Sub-sampling the history reduces redundancy and keeps the compute light. If the history is sub-sampled too coarsely, however, important but brief events may be dropped before the learned selector can examine them. So we designed experiments to test how densely the history needs to be sampled, at a memory budget of 64 tokens.
Success rate (%): columns Counting to AVG; Cost at deployment: columns Stored tokens to Added compute (TFLOPs).
| Sub-sampling | Counting | Permanence | Reference | Imitation | AVG | Stored tokens | Peak storage (MB) | Added compute (TFLOPs) |
|---|---|---|---|---|---|---|---|---|
| Pool, | 58.44 | 21.33 | 26.00 | 34.56 | 35.08 | 161 | 0.03 | |
| Pool, | 59.17 | 25.17 | 24.56 | 38.67 | 36.89 | 161 | 0.05 | |
| Pool, | 75.44 | 23.17 | 54.67 | 66.00 | 54.82 | 161 | 0.11 | |
| Pool, | 73.11 | 24.94 | 51.67 | 59.72 | 52.36 | 161 | 0.17 | |
| Stride 6, growing tree | 73.17 | 23.83 | 49.83 | 53.44 | 50.07 | 17 | 0.51 | |
| Stride 2, growing tree | 72.67 | 22.67 | 53.00 | 58.33 | 51.67 | 67 | 1.90 |
Stored tokens (2,048-dimensional, 16-bit each) is the most a variant needs after observed frames, and peak storage is its maximum over the longest possible evaluation episode (, video included). Added compute is per query and relative to without memory (2.92 TFLOPs per query). It is independent of and excludes the encoding of new frames (3.54 TFLOPs per query for every memory method).
The key observations are,
- Temporal coverage matters for the selector to work well. For both the candidate pool and the stride, covering more frames (pool size from 128 to 512 tokens, and stride from 6 to 2) generally gives a higher success rate.
- The memory needs to keep the right historical evidence, instead of a very large budget. With a memory budget of only 64 tokens, D&R reaches a success rate of 54.82, against 32.22 for the best baseline (HAMLET) at the same budget. It is also competitive with D&R at a memory budget of 512 tokens (58.54, the third place on the RoboMME leaderboard). This tells us that the most important thing is to spend the context in the right place.
- Storage. The candidate pool is re-drawn from the whole history at every step, so it stores the tokens of every observed frame, and its storage grows linearly with the episode length ( tokens, 161 MB for the longest episode). The growing tree stores only the frames that enter it, up to a fixed capacity, so its storage is bounded (17 MB at stride 6 and 67 MB at stride 2).
- Compute. The added compute does not grow with the episode length for any variant. The candidate pool adds 0.03 to 0.17 TFLOPs to the 2.92 TFLOPs of , and the growing tree adds 0.51 TFLOPs at stride 6 and 1.90 TFLOPs at stride 2. The pool of 512 tokens gives the highest success rate at a small compute cost, so we use it for the main results.
Offline-online refers to the growing history. At deployment the history grows step by step, and D&R builds the same tree for the same observed history as in training. To make this clearer, we refined the opening of the method section: "This section describes how we maximise the conditional mutual information in practice. We optimise the memory over the original history tokens, which turns it into a discrete optimisation problem, namely the selection of a subset of the history . We estimate the mapping with a parametric family built from a selector with parameters . However, the history grows step by step, whereas a parametric selector has a fixed input size and should stay lightweight. Section 5.1 casts memory as budgeted discrete selection and introduces D&R, which composes the selection over a history of any length recursively from a fixed-size selector; Section 5.2 describes the selector and its training."
Response to Reviewer fS2a (3/3)
R3 to W1: the real-robot configuration.
We agree. With and , Algorithm 1 requires only one -to- selection, so the real-robot experiments support learned selection and do not validate the hierarchical recursion. The recursion is validated in simulation. With the same candidate pool and budget, the recursive selector reaches 54.82 against 42.36 for the flat selector (the ablation above).
R4 to W2 and Q3: data for the experimental results.
Our up-to-date best result (we fixed the data loading of D&R after the submission), D&R at a budget of 512 tokens (58.5%), ranks third on the RoboMME leaderboard. D&R with a memory budget of only 64 tokens (54.8%) is competitive with it. This goes back to our motivation that the key to solving the memory problem is finding the key historical fact that the current observation lacks, and it shows that even a very limited memory budget is enough to perform well.
All the data for the experimental results in the paper (per method, per checkpoint and per seed) can be found at https://anonymous.4open.science/r/Divide-and-Remember-6363/. We report the average and std of all methods over the 80k, 90k and 100k checkpoints and seeds 0/7/42. In Tables 10 and 11 of the revision (Tables 8 and 9 of the submission), the results taken from the RoboMME paper (Dai et al., 2026) and from the T$^2$Mem paper (Liu et al., 2026) are marked and , respectively. The checkpoints will also be released after the double-blind review.
R5 to W3 and Q4: baseline adaptation.
We thank the reviewer for the suggestion, and we have added the details to Appendix C.1 (Experimental setup). As the reviewer notes, a controlled common-backbone comparison is reasonable. Our results compare all memory methods within a shared setup, and all memories are modulated into the action expert in the same way. What we care about is what each memory method keeps within its memory budget. The added paragraph reads:
[...] For HAMLET we use the authors' open-source code and only adapt their GR00T backbone to , keeping the time-contrastive initialisation and the moment-token and history settings of the original. For RB-VLA we re-implement the recursive belief update on and train it with the world-model objective of the original. We do not include its action- and proprioception-conditioned belief estimation, since proprioception is excluded from all inputs, nor its episodic VLM querying, since every method here runs the VLM at every step. For the other methods we use the code released with RoboMME (Dai et al., 2026). All code will be released after the double-blind review.
The efficiency comparison concerns our adaptation, not the original RB-VLA. In our setup every method runs the VLM at every step, so the training time of 12.5 days is for this adaptation, and we do not claim that the original RB-VLA requires it.
On whether latent memory loses spatial detail, we added a controlled comparison that does not depend on the adapted baselines. D&R+latent keeps the selector, the candidate pool, the budget and the training recipe of D&R, and only fuses the selected tokens into latent tokens of the same size. This alone lowers the average success rate from 54.82 to 28.93, and the imitation suite falls from 66.0 to 16.3 (Appendix C.4.3, pages 31-32, excerpt).
Response to Reviewer fS2a (1/3)
Thank you for the reviewer's comments. We especially appreciate the suggestion on the ablation design, which inspired us to give a full picture of how different token-selection rules affect performance. We greatly appreciate your recognition of our method's contribution, and we believe that filling the gap between fixed frame-selection heuristics and learned latent compression is meaningful for this line of research. Thank you also for acknowledging our experimental work. We have improved our manuscript according to your comments and uploaded the revised version. We hope our clarifications will convince you to reconsider the score. Please let us know if further clarification is required.
R1 to W1 and Q2: ablation of recursion and parameter sharing, and different token pruning.
We thank the reviewer for the suggestion and ran the controls as specified. All seven rows use the same 512 candidate tokens, the same 64 output tokens and the same positional information. They differ in the rule that picks the 64 tokens: (1) fixed rules (uniform or random patch selection from the identical candidate pool) and (2) learned selectors (flat and recursive). Uniform selection uses two hand-picked patch pairs: 5, 6 (where the learned selector concentrates) and 13, 14 (mostly background).
The table separates temporal coverage, learned selection and recursion, and gives the following key observations:
Detailed descriptions of the rules, illustrations of the selectors and the full results are in the newly added Appendix C.2 ("Ablation: token-selection rule", pages 24-26, excerpt).
R2 to W1 and Q1: how the history is maintained at deployment.
Thank you for the question. We added Appendix B, "Details of Divide-and-Remember" (pages 20-22, excerpt), to detail and illustrate how D&R builds the recursive tree as the history grows, with either a growing tree or a fixed tree.
(1) Growing tree. The growing history streams into the D&R selector and the recursive tree grows naturally with it. To further bound the compute, the history can be sub-sampled with a stride before it enters the tree. Only every s-th frame of the history enters the tree as it streams in.