acceptodds
Under review as a conference paper at ICLR 2027

Divide-and-Remember: Recursive Action-Relevant Memory for Long-Horizon VLA Policies

Abstract

Vision-language-action (VLA) models struggle on history-dependent manipulation tasks, where the current observation alone does not determine the action, and the policy needs a memory of the history. Existing memory methods decide what to remember by design, for example, keeping frames with large pixel changes, and show inconsistent gains across tasks. We view what to remember as an optimisation problem. From the POMDP formulation of imitation learning, we show that the optimal memory maximises the conditional mutual information between the action and the memory given the current observation, the information the decision needs but the observation lacks. We propose Divide-and-Remember (D&R), a recursive memory method that learns to maximise this objective, scaling to long contexts while staying compute-light. It involves two strategies: (1) the selection over the full history is divided recursively into subproblems of top- selection over tokens, so that fixed-size, lightweight selectors learned end-to-end support an unbounded history, and the composition comes with an approximation guarantee; (2) all recursion blocks share one selector, which captures the selection rule common to every block and keeps efficient. On RoboMME, a benchmark of long-horizon manipulation tasks that require remembering when, where, what, and how to act, D&R achieves a state-of-the-art average success rate with consistent gains across all four suites under a budget of only tokens; real-robot experiments show the same gain.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot manipulation, memory, partial observability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fS2aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Divide-and-Remember (D&R), a learned visual-memory selection method for history-dependent vision–language–action policies. It motivates memory learning through the conditional mutual information (I(a_t;m_t\mid o_t)), which measures the action-relevant information retained beyond the current observation. D&R recursively applies a shared selector that reduces up to (2K) candidate tokens to (K), preserving selected visual features instead of replacing them with newly synthesized latent summaries. The selector is trained jointly with a π₀.₅ policy using its flow-matching action loss.

On RoboMME, D&R reports 38.58% average success with 64 memory tokens, compared with 32.22% for the adapted HAMLET baseline and 27.93% for FrameSamp. Additional experiments vary the candidate pool and memory budget. Four real-robot tasks yield 35/40 successful trials, compared with 22/40 for FrameSamp.

Critical strengths
  1. Learning discrete visual-memory selection is a useful contribution with a clear practical motivation.

The method occupies a meaningful position between fixed frame-selection heuristics and learned latent compression. It learns selection from the control objective while preserving the values of selected visual features. Sharing a small selector across a reduction tree is a coherent way to accommodate varying candidate-set sizes without expanding the parameter count.

  1. The reported gains are substantial in a relevant evaluation setting and are supported by real-robot evidence.

RoboMME directly tests history dependence, making it well suited to the paper’s central question. Under the reported 64-token setting, D&R improves average success over HAMLET by 6.36 percentage points and over FrameSamp by 10.65 points. The improvement is especially pronounced on Imitation: 37.83% versus 19.50% and 22.06%, respectively. The manuscript also acknowledges cases where latent memory remains competitive, which helps delimit the contribution. The real-robot results provide useful additional evidence: D&R succeeds in 35/40 trials while using 1,024 output memory tokens, whereas FrameSamp succeeds in 22/40 with 2,048 tokens.

Critical weaknesses
  1. The experiments do not isolate the recursive design, and the implemented history pipeline does not support the claim of full-history memory.

Sections 5.1–5.2 formulate selection over the full history and emphasize an unbounded-history construction with consistent offline and online behavior. However, Section 6.3 reveals that the evaluated selector receives a uniformly sampled candidate pool: the main configuration contains 32 frames with 16 pooled tokens each. Consequently, an event excluded by this initial sampling cannot be recovered by the learned selector. This leaves unresolved whether the deployed system maintains a persistent recursive memory or repeatedly selects from an archive of sampled historical frames. These alternatives have different storage requirements, update costs, and failure modes.

The ablations vary candidate-pool size, output budget, token types, and auxiliary loss, but do not compare the recursive selector with a flat learned selector, or with uniform/random patch selection from the identical candidate pool. At (K=64), FrameSamp retains four complete pooled frames, while D&R can distribute its output across 32 candidate frames. Thus, the existing comparison does not determine how much improvement comes from learned selection, broader temporal coverage, or the recursive organization.

Furthermore, the real-robot configuration uses (|H|=2048) and (K=1024). Under Algorithm 1, this requires only one (2K)-to-(K) selection, so these experiments support learned selection but do not validate hierarchical recursion. A matched selector ablation and an explicit online update/storage protocol are important because recursion and long-history scalability are central contributions, not peripheral implementation choices.

  1. Inconsistent reporting metrics. The stated evaluation protocol is 50 episodes per task, averaged over three checkpoints and three seeds. Under that protocol, an equally weighted per-task mean must occur in increments of (100/450≈0.2222100/450\approx0.2222) percentage points. Several entries do not follow this rule, including D&R’s 7.67% on InsertPeg in Table 9 and D&R+pred’s 41.25% on BinFill in Table 6. This mean and standard deviation cannot describe three or nine nonnegative run-level success rates as stated. Table captions also alternate between variation “over seeds” and “over evaluation runs.” These may be transcription errors or undocumented protocol exceptions, but they require reconciliation with per-run success counts.

  2. Insufficiently specified baseline adaptations HAMLET’s original method includes time-contrastive initialization and specific moment-token/history settings, whose treatment in this implementation is not documented. The original RB-VLA uses action/proprioception-conditioned belief estimation and largely episodic VLM querying, whereas this paper excludes proprioception and imposes a common π₀.₅ integration. A controlled common-backbone comparison is reasonable, but its results characterize those adapted implementations. They do not, without further evidence, establish that latent memory inherently loses the relevant spatial detail or that RB-VLA generally requires the reported computational cost. HAMLET, RB-VLA.

Questions for authors

Q1 Implementation clarification. At deployment, is the candidate pool reconstructed by sampling an archive of historical frames, or maintained incrementally through persistent tree nodes? Please specify details retained between policy calls, the frame-resampling rule, and how storage and update latency scale with episode length. Plus, how does the evaluated implementation realize the offline–online equivalence claimed in Section 5.1?

Q2 Evidence needed to validate the central contribution. With the same 512 candidate tokens, 64 output tokens, positional information, and policy-training recipe, how does D&R compare with uniform/random patch selection and a flat learned top-(K) selector? These controls would distinguish temporal coverage and learned selection from the additional value of recursion.

Q3 Reporting clarification. Please provide per-checkpoint, per-seed success counts and denominators for Tables 6–9, explain how standard deviations are computed, and reconcile the entries identified in W2. Were any variants evaluated using different episode counts or fewer runs?

Q4 Baseline clarification. Was HAMLET’s time-contrastive initialization retained, and what were its history length and moment-token allocation? Which components and objectives of RB-VLA were preserved? Please distinguish changes required for π₀.₅ integration from optional simplifications, and clarify whether the efficiency comparison concerns the original method or this adaptation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fS2a (1/3)

Thank you for the reviewer's comments. We especially appreciate the suggestion on the ablation design, which inspired us to give a full picture of how different token-selection rules affect performance. We greatly appreciate your recognition of our method's contribution, and we believe that filling the gap between fixed frame-selection heuristics and learned latent compression is meaningful for this line of research. Thank you also for acknowledging our experimental work. We have improved our manuscript according to your comments and uploaded the revised version. We hope our clarifications will convince you to reconsider the score. Please let us know if further clarification is required.

R1 to W1 and Q2: ablation of recursion and parameter sharing, and different token pruning.

We thank the reviewer for the suggestion and ran the controls as specified. All seven rows use the same 512 candidate tokens, the same 64 output tokens and the same positional information. They differ in the rule that picks the 64 tokens: (1) fixed rules (uniform or random patch selection from the identical candidate pool) and (2) learned selectors (flat and recursive). Uniform selection uses two hand-picked patch pairs: 5, 6 (where the learned selector concentrates) and 13, 14 (mostly background).

SelectorContrastOverall success rate (%)
Fixed rule (no learning)
FrameSampReference.28.61
RandomTemporal coverage (vs. FrameSamp).25.11
Uniform (indices 5, 6)Even coverage of all frames (vs. FrameSamp) and what to keep (vs. Random).43.58
Uniform (indices 13, 14)What to keep (vs. Random and Uniform, indices 5, 6).25.67
D&R (learned)
flatLearning what to keep (vs. Random and Uniform) and recursion (vs. recursive).42.36
recursive (w/o parameter sharing)Parameter sharing (vs. recursive).35.06
recursiveRecursion (vs. flat).54.82

The table separates temporal coverage, learned selection and recursion, and gives the following key observations:

  1. Temporal coverage alone does not help. Covering all 32 frames with random patches lowers the success rate from 28.61 (FrameSamp) to 25.11. Covering them with a fixed pair that mostly shows background (indices 13, 14) gives 25.67, which is also below FrameSamp.
  2. Which patches are kept matters. Covering the same 32 frames with the fixed pair 5, 6, where the learned selector concentrates, raises the success rate from 28.61 to 43.58 (+15.0).
  3. Learned selection recovers the hand-picked rule and, in a single cut, does not exceed it. The flat selector reaches 42.36 without being told where to look, within 1.2 points of the pair that we picked by inspecting the learned selector.
  4. Recursion. With the same pool and budget, the recursive selector reaches 54.82 against 42.36 for the flat selector (+12.5). Each level of the tree re-scores only the tokens kept by the level below, so the least important tokens, such as background, are dropped at the leaves. The recursive structure also reduces the cost. The flat selector attends over all 512 tokens at once (5122≈2.6×105512^2\approx2.6\times10^5 token pairs per layer), whereas the tree makes seven calls on 128 tokens (7×1282≈1.1×1057\times128^2\approx1.1\times10^5). The cost of the flat selector grows quadratically with the pool, and that of the tree linearly.
  5. Parameter sharing. One selector shared by all nodes of the tree has a third of the parameters of a separate selector per level (25.2M against 75.6M), and it reaches a higher success rate (54.82 against 35.06).

Detailed descriptions of the rules, illustrations of the selectors and the full results are in the newly added Appendix C.2 ("Ablation: token-selection rule", pages 24-26, excerpt).

R2 to W1 and Q1: how the history is maintained at deployment.

Thank you for the question. We added Appendix B, "Details of Divide-and-Remember" (pages 20-22, excerpt), to detail and illustrate how D&R builds the recursive tree as the history grows, with either a growing tree or a fixed tree.

(1) Growing tree. The growing history streams into the D&R selector and the recursive tree grows naturally with it. To further bound the compute, the history can be sub-sampled with a stride before it enters the tree. Only every ss-th frame of the history enters the tree as it streams in.

AuthorsOpenReview ↗

Response to Reviewer fS2a (2/3)

R2 (continued).

(2) Fixed tree (the original main results). As the history grows, we sub-sample it into a pool of fixed size, so that we only maintain a tree of fixed size with a constant number of leaves. Specifically, a fixed number of frames is sampled uniformly from the history observed so far, which gives a candidate pool of fixed size ∣H∣\vert\mathcal{H}\vert. The pool is re-drawn at every step, so the spacing between its frames widens as the history grows, while the cost of the selector stays independent of the history length.

Sub-sampling the history reduces redundancy and keeps the compute light. If the history is sub-sampled too coarsely, however, important but brief events may be dropped before the learned selector can examine them. So we designed experiments to test how densely the history needs to be sampled, at a memory budget of 64 tokens.

Success rate (%): columns Counting to AVG; Cost at deployment: columns Stored tokens to Added compute (TFLOPs).

Sub-samplingCountingPermanenceReferenceImitationAVGStored tokensPeak storage (MB)Added compute (TFLOPs)
Pool, ∣H∣=128\vert\mathcal{H}\vert=12858.4421.3326.0034.5635.0816T16T1610.03
Pool, ∣H∣=256\vert\mathcal{H}\vert=25659.1725.1724.5638.6736.8916T16T1610.05
Pool, ∣H∣=512\vert\mathcal{H}\vert=51275.4423.1754.6766.0054.8216T16T1610.11
Pool, ∣H∣=1024\vert\mathcal{H}\vert=102473.1124.9451.6759.7252.3616T16T1610.17
Stride 6, growing tree73.1723.8349.8353.4450.07min⁡(16T/6,4,096)\min(16T/6, 4{,}096)170.51
Stride 2, growing tree72.6722.6753.0058.3351.67min⁡(16T/2,16,384)\min(16T/2, 16{,}384)671.90

Stored tokens (2,048-dimensional, 16-bit each) is the most a variant needs after TT observed frames, and peak storage is its maximum over the longest possible evaluation episode (T=2,456T=2{,}456, video included). Added compute is per query and relative to π0.5\pi_{0.5} without memory (2.92 TFLOPs per query). It is independent of TT and excludes the encoding of new frames (3.54 TFLOPs per query for every memory method).

The key observations are,

  1. Temporal coverage matters for the selector to work well. For both the candidate pool and the stride, covering more frames (pool size from 128 to 512 tokens, and stride from 6 to 2) generally gives a higher success rate.
  2. The memory needs to keep the right historical evidence, instead of a very large budget. With a memory budget of only 64 tokens, D&R reaches a success rate of 54.82, against 32.22 for the best baseline (HAMLET) at the same budget. It is also competitive with D&R at a memory budget of 512 tokens (58.54, the third place on the RoboMME leaderboard). This tells us that the most important thing is to spend the context in the right place.
  3. Storage. The candidate pool is re-drawn from the whole history at every step, so it stores the tokens of every observed frame, and its storage grows linearly with the episode length (16T16T tokens, 161 MB for the longest episode). The growing tree stores only the frames that enter it, up to a fixed capacity, so its storage is bounded (17 MB at stride 6 and 67 MB at stride 2).
  4. Compute. The added compute does not grow with the episode length for any variant. The candidate pool adds 0.03 to 0.17 TFLOPs to the 2.92 TFLOPs of π0.5\pi_{0.5}, and the growing tree adds 0.51 TFLOPs at stride 6 and 1.90 TFLOPs at stride 2. The pool of 512 tokens gives the highest success rate at a small compute cost, so we use it for the main results.

Offline-online refers to the growing history. At deployment the history grows step by step, and D&R builds the same tree for the same observed history as in training. To make this clearer, we refined the opening of the method section: "This section describes how we maximise the conditional mutual information I(at;mt∣ot)I(a_t;m_t\mid o_t) in practice. We optimise the memory mt=M(ht)m_t=\mathcal M(h_t) over the original history tokens, which turns it into a discrete optimisation problem, namely the selection of a subset of the history hth_t. We estimate the mapping M\mathcal M with a parametric family Mψ\mathcal M_\psi built from a selector with parameters ψ\psi. However, the history grows step by step, whereas a parametric selector has a fixed input size and should stay lightweight. Section 5.1 casts memory as budgeted discrete selection and introduces D&R, which composes the selection over a history of any length recursively from a fixed-size selector; Section 5.2 describes the selector and its training."

AuthorsOpenReview ↗

Response to Reviewer fS2a (3/3)

R3 to W1: the real-robot configuration.

We agree. With ∣H∣=2048\vert\mathcal{H}\vert=2048 and K=1024K=1024, Algorithm 1 requires only one 2K2K-to-KK selection, so the real-robot experiments support learned selection and do not validate the hierarchical recursion. The recursion is validated in simulation. With the same candidate pool and budget, the recursive selector reaches 54.82 against 42.36 for the flat selector (the ablation above).

R4 to W2 and Q3: data for the experimental results.

Our up-to-date best result (we fixed the data loading of D&R after the submission), D&R at a budget of 512 tokens (58.5%), ranks third on the RoboMME leaderboard. D&R with a memory budget of only 64 tokens (54.8%) is competitive with it. This goes back to our motivation that the key to solving the memory problem is finding the key historical fact that the current observation lacks, and it shows that even a very limited memory budget is enough to perform well.

All the data for the experimental results in the paper (per method, per checkpoint and per seed) can be found at https://anonymous.4open.science/r/Divide-and-Remember-6363/. We report the average and std of all methods over the 80k, 90k and 100k checkpoints and seeds 0/7/42. In Tables 10 and 11 of the revision (Tables 8 and 9 of the submission), the results taken from the RoboMME paper (Dai et al., 2026) and from the T$^2$Mem paper (Liu et al., 2026) are marked †^\dagger and ‡^\ddagger, respectively. The checkpoints will also be released after the double-blind review.

R5 to W3 and Q4: baseline adaptation.

We thank the reviewer for the suggestion, and we have added the details to Appendix C.1 (Experimental setup). As the reviewer notes, a controlled common-backbone comparison is reasonable. Our results compare all memory methods within a shared π0.5\pi_{0.5} setup, and all memories are modulated into the action expert in the same way. What we care about is what each memory method keeps within its memory budget. The added paragraph reads:

[...] For HAMLET we use the authors' open-source code and only adapt their GR00T backbone to π0.5\pi_{0.5}, keeping the time-contrastive initialisation and the moment-token and history settings of the original. For RB-VLA we re-implement the recursive belief update on π0.5\pi_{0.5} and train it with the world-model objective of the original. We do not include its action- and proprioception-conditioned belief estimation, since proprioception is excluded from all inputs, nor its episodic VLM querying, since every method here runs the VLM at every step. For the other methods we use the code released with RoboMME (Dai et al., 2026). All code will be released after the double-blind review.

The efficiency comparison concerns our π0.5\pi_{0.5} adaptation, not the original RB-VLA. In our setup every method runs the VLM at every step, so the training time of 12.5 days is for this adaptation, and we do not claim that the original RB-VLA requires it.

On whether latent memory loses spatial detail, we added a controlled comparison that does not depend on the adapted baselines. D&R+latent keeps the selector, the candidate pool, the budget and the training recipe of D&R, and only fuses the selected tokens into latent tokens of the same size. This alone lowers the average success rate from 54.82 to 28.93, and the imitation suite falls from 66.0 to 16.3 (Appendix C.4.3, pages 31-32, excerpt).

Reviewer_b7msOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Divide-and-Remember (D&R), a recursive memory mechanism for long-horizon VLA policies. It uses a shared, action-supervised selector to retain relevant historical visual tokens under a fixed memory budget. D&R achieves 38.6% average success on RoboMME and shows promising results on real-robot tasks.

Critical strengths
  1. Clear motivation and practical method.

The paper addresses an important problem in long-horizon manipulation: deciding what information to retain from history. Learning to select action-relevant tokens instead of relying on fixed heuristics is a reasonable approach. The recursive design is also lightweight and relatively straightforward.

  1. Strong empirical results.

D&R outperforms several memory baselines on RoboMME, with useful analyses across different memory demands. The real-robot results are encouraging, achieving 35/40 successes compared with 22/40 for FrameSamp. These experiments provide meaningful evidence of the method's practical value.

Critical weaknesses
  1. The contributions of the main design choices are not sufficiently isolated.

The paper emphasizes recursive selection and parameter sharing, but neither is directly ablated. I would like to see a comparison with a non-recursive learned selector using the same candidate pool and memory budget. Without this, it is difficult to tell how much of the improvement comes from recursion rather than simply learning to select tokens through the action loss.

I also think the motivation for retaining raw visual tokens needs stronger evidence. Although the paper compares against HAMLET and RB-VLA, these methods differ in both memory representation and construction. A controlled comparison that compresses the same selected historical tokens into learned latent representations would help establish whether preserving the original visual-token space is actually responsible for the gains.

  1. The scalability claim needs further clarification.

While the recursive formulation can handle increasingly long inputs, the actual experiments first uniformly sample a fixed-size candidate pool from the history. Important but brief events may therefore be missed before the learned selector can examine them. It is also unclear how the full historical pool is maintained online and whether its storage and update costs are included in the efficiency analysis.

The mutual-information formulation provides useful motivation, but it does not directly establish the advantage of the particular recursive selection procedure.

Questions for authors
  1. Could the authors compare D&R with a non-recursive learned selector using the same candidate pool, memory budget, and comparable computation? An ablation of parameter sharing across recursion levels would also help clarify the contribution of the proposed architecture.

  2. The paper argues that retaining original visual tokens helps preserve spatial information. Although HAMLET and RB-VLA are included as baselines, they also use different memory construction mechanisms. Could the authors isolate the benefit of raw-token representations, for example by comparing D&R with a variant that fuses the same selected historical tokens into learned latent representations?

  3. Section 6.3 describes constructing a fixed-size candidate pool by uniformly sampling historical frames. Could the authors clarify how this pool is maintained during online execution, whether its storage and update costs are included in the efficiency analysis, and how the sampling strategy handles brief but important historical events?

5 comments
AuthorsOpenReview ↗

Response to Reviewer b7ms (1/3)

Thank you for the reviewer's comments and for your recognition of our work. We appreciate your acknowledgement of our presentation, method and experimental results. We also appreciate your suggestions, especially the one on the scalability of the recursive formulation of our method. We have improved our manuscript according to your comments and uploaded the revised version. We hope our clarifications and additional experiments will convince you to consider raising the score. Please let us know if further clarification is required.

R1 to W1 and Q1: ablation of recursion and parameter sharing.

We thank the reviewer for the suggestion and ran the ablations. All rows use the same 512 candidate tokens, the same 64 output tokens and the same positional information.

SelectorContrastOverall success rate (%)
D&R (learned)
flatReference.42.36
recursive (w/o parameter sharing)Parameter sharing (vs. recursive).35.06
recursiveRecursion (vs. flat).54.82

The table separates recursion and parameter sharing, and gives the following key observations:

  1. Recursion. With the same pool and budget, the recursive selector reaches 54.82 against 42.36 for the flat selector (+12.5). Each level of the tree re-scores only the tokens kept by the level below, so the least important tokens, such as background, are dropped at the leaves. The recursive structure also reduces the cost. The flat selector attends over all 512 tokens at once (5122≈2.6×105512^2\approx2.6\times10^5 token pairs per layer), whereas the tree makes seven calls on 128 tokens (7×1282≈1.1×1057\times128^2\approx1.1\times10^5). The cost of the flat selector grows quadratically with the pool, and that of the tree linearly.
  2. Parameter sharing. One selector shared by all nodes of the tree has a third of the parameters of a separate selector per level (25.2M against 75.6M), and it reaches a higher success rate (54.82 against 35.06).

Detailed descriptions of the rules, illustrations of the selectors and the full results are in the newly added Appendix C.2 ("Ablation: token-selection rule", pages 24-26, excerpt).

R2 to W1 and Q2: isolating the benefit of raw visual tokens over latent memory.

We thank the reviewer for the thoughtful suggestion. To give a clearer view of what the raw-token representation contributes, we add a variant, D&R+latent, in which the selected tokens are fused into learned latents while the selector, the candidate pool, the budget, the positional information and the training recipe stay the same as in D&R.

The selected visual tokens are fused by a Perceiver Resampler inserted between the selector and the policy of D&R. The Perceiver Resampler module is commonly attached to the vision encoder to learn a fixed-size latent embedding of the visual features before it is passed into the cross-attention layer (Flamingo, Alayrac et al., 2022; BLIP-2, Li et al., 2023). Only fusing the selected visual tokens into a latent representation of the same size lowers the average success rate from 54.82 to 28.93 (-25.9 points), and the imitation suite, which depends most on spatial detail, drops the most. The observation is therefore consistent with the comparison with HAMLET and RB-VLA: preserving the original visual-token space is what preserves the spatial detail.

The full result and a more detailed analysis are given in the new Appendix C.4.3, "Ablation: historical token or latent token" (pages 31-32, excerpt), of the updated manuscript.

AuthorsOpenReview ↗

Response to Reviewer b7ms (2/3)

R3 to W2 and Q3: the recursive formulation handles increasingly long inputs, with a growing tree or a candidate pool.

Thank you for your suggestion. We added an experiment to test the scalability of D&R, that is, whether the recursive formulation can handle increasingly long inputs. We added Appendix B, "Details of Divide-and-Remember" (pages 20-22, excerpt), to detail and illustrate how D&R builds the recursive tree as the history grows, with either a growing tree or a fixed tree.

(1) Growing tree. The growing history streams into the D&R selector and the recursive tree grows naturally with it. To further bound the compute, the history can be sub-sampled with a stride before it enters the tree. Only every ss-th frame of the history enters the tree as it streams in.

(2) Fixed tree (the original main results). As the history grows, we sub-sample it into a pool of fixed size, so that we only maintain a tree of fixed size with a constant number of leaves. Specifically, a fixed number of frames is sampled uniformly from the history observed so far, which gives a candidate pool of fixed size ∣H∣\vert\mathcal{H}\vert. The pool is re-drawn at every step, so the spacing between its frames widens as the history grows, while the cost of the selector stays independent of the history length.

Sub-sampling the history reduces redundancy and keeps the compute light. If the history is sub-sampled too coarsely, however, important but brief events may be dropped before the learned selector can examine them. So we designed experiments to test how densely the history needs to be sampled, at a memory budget of 64 tokens.

Success rate (%): columns Counting to AVG; Cost at deployment: columns Stored tokens to Added compute (TFLOPs).

Sub-samplingCountingPermanenceReferenceImitationAVGStored tokensPeak storage (MB)Added compute (TFLOPs)
Pool, ∣H∣=128\vert\mathcal{H}\vert=12858.4421.3326.0034.5635.0816T16T1610.03
Pool, ∣H∣=256\vert\mathcal{H}\vert=25659.1725.1724.5638.6736.8916T16T1610.05
Pool, ∣H∣=512\vert\mathcal{H}\vert=51275.4423.1754.6766.0054.8216T16T1610.11
Pool, ∣H∣=1024\vert\mathcal{H}\vert=102473.1124.9451.6759.7252.3616T16T1610.17
Stride 6, growing tree73.1723.8349.8353.4450.07min⁡(16T/6,4,096)\min(16T/6, 4{,}096)170.51
Stride 2, growing tree72.6722.6753.0058.3351.67min⁡(16T/2,16,384)\min(16T/2, 16{,}384)671.90

The key observations are,

  1. Temporal coverage matters for the selector to work well. For both the candidate pool and the stride, covering more frames (pool size from 128 to 512 tokens, and stride from 6 to 2) generally gives a higher success rate.
  2. The memory needs to keep the right historical evidence, instead of a very large budget. With a memory budget of only 64 tokens, D&R reaches a success rate of 54.82, against 32.22 for the best baseline (HAMLET) at the same budget. It is also competitive with D&R at a memory budget of 512 tokens (58.54, the third place on the RoboMME leaderboard). This tells us that the most important thing is to spend the context in the right place.
  3. Storage. The candidate pool is re-drawn from the whole history at every step, so it stores the tokens of every observed frame, and its storage grows linearly with the episode length (16T16T tokens, 161 MB for the longest episode). The growing tree stores only the frames that enter it, up to a fixed capacity, so its storage is bounded (17 MB at stride 6 and 67 MB at stride 2).
  4. Compute. The added compute does not grow with the episode length for any variant. The candidate pool adds 0.03 to 0.17 TFLOPs to the 2.92 TFLOPs of π0.5\pi_{0.5}, and the growing tree adds 0.51 TFLOPs at stride 6 and 1.90 TFLOPs at stride 2. The pool of 512 tokens gives the highest success rate at a small compute cost, so we use it for the main results.

The efficiency analysis in Figure 6(a) already includes the selector. Its horizontal axis is the TFLOPs of one forward pass, which covers the selector, the VLM and the action expert with 10 denoising steps.

The densest variant, stride 2, does not exceed the pool of 512 tokens, which suggests that missed brief events are not what limits the pool on these tasks.

AuthorsOpenReview ↗

Response to Reviewer b7ms (3/3)

R4 to W2: the connection between conditional mutual information maximization (what the theoretical objective is) and D&R (how we optimise it in practice).

D&R gives a way to optimise the theoretical objective. It formulates the objective as a budgeted discrete optimisation problem, selecting original history tokens (for token fidelity) under a budget, over a history whose length grows with the episode. Empirical evidence also shows that this buys back the spatial detail that latent memories optimised through action supervision lose. We believe that filling the gap between fixed frame-selection heuristics and learned latent compression is meaningful for this line of research.

The recursion is how we solve this discrete problem over a growing history with a selector of fixed input size. Whether the recursion helps in practice is an empirical question, and we test it in the ablation above. With the same candidate pool and budget, the recursive selector reaches 54.82 against 42.36 for the flat selector.

To help readers better understand our motivation, we refined the opening of the method section: "This section describes how we maximise the conditional mutual information I(at;mt∣ot)I(a_t;m_t\mid o_t) in practice. We optimise the memory mt=M(ht)m_t=\mathcal M(h_t) over the original history tokens, which turns it into a discrete optimisation problem, namely the selection of a subset of the history hth_t. We estimate the mapping M\mathcal M with a parametric family Mψ\mathcal M_\psi built from a selector with parameters ψ\psi. However, the history grows step by step, whereas a parametric selector has a fixed input size and should stay lightweight. Section 5.1 casts memory as budgeted discrete selection and introduces D&R, which composes the selection over a history of any length recursively from a fixed-size selector; Section 5.2 describes the selector and its training."

Reviewer_xY4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Divide-and-Remember (D&R), a memory mechanism for history-dependent VLA policies. A shared selector recursively retains historical visual tokens under a fixed memory budget and is trained jointly with the policy using an action loss. The method is motivated by a conditional mutual information objective and improves average success on RoboMME, with additional validation on four real-world tasks.

Critical strengths
  • D&R achieves promising empirical results: under a 64-token budget, it reaches 38.6% average success on RoboMME, compared with 32.2% for HAMLET.
  • The shared recursive selector provides a practical architecture for learning which historical tokens to retain while keeping the selector’s input size bounded.
  • The evaluation covers several memory demands and includes real-world experiments, budget ablations, and memory visualizations.
Critical weaknesses
  • The connection between the theoretical objective and practical training is insufficiently established. The mutual information decomposition motivates maximizing conditional action likelihood, whereas the implementation uses flow-matching loss. The paper provides no further derivation or evidence connecting its optimization to increased conditional mutual information. Consequently, the analysis appears largely independent of the proposed algorithm, substantially weakening the claimed theoretical contribution.

  • The analysis does not specifically justify D&R. The likelihood-based argument also applies to other memory modules trained through action supervision. It neither motivates the particular recursive selection structure nor establishes its optimality. Moreover, policy sufficiency requires zero residual information, which need not be achievable under a fixed token budget.

  • Related-work coverage is incomplete. The authors should analyze and discuss NativeMEM (Wang et al., 2026), which compresses historical frames using the VLA’s own vision encoder, and Gated Memory Policy (Gao et al., 2026), which learns when and what to recall through gating and cross-attention. These works are directly relevant to pretrained representation compatibility and learned memory access. The discussion should clarify D&R’s distinctions in memory representation, training objectives, and computational trade-offs, rather than broadly characterizing prior memory designs as hand-crafted.

  • Long-history scalability is insufficiently validated. Although the recursive architecture can accommodate variable-length inputs, the experiments operate on bounded candidate pools uniformly sampled from history. The paper does not evaluate how memory accuracy and online storage/update costs scale with increasing episode length, or clarify how candidate resampling interacts with recursive caching. Thus, the practical benefits of supporting an “unbounded history” remain insufficiently demonstrated.

  • The evaluation provides limited evidence for complex long-horizon memory. Although PutBottles reportedly lasts up to four minutes, the demonstrated memory demands involve only a few items. Episode duration alone does not establish memory complexity. Tasks requiring more remembered entities, longer ordered sequences, or multiple interdependent historical facts are needed to substantiate the long-horizon claims.

Questions for authors
  • Can you establish a formal connection between the flow-matching objective and conditional mutual information, or narrow the claim to an information-theoretic motivation?
  • How much improvement comes specifically from recursion compared with a learned flat selector under the same candidate pool and memory budget?

I would be willing to raise my score if the authors establish a convincing connection between conditional mutual information maximization, the actual flow-matching training objective, and the proposed recursive memory design.

5 comments
AuthorsOpenReview ↗

Response to Reviewer xY4U (1/4)

Thank you for the reviewer's comments, especially for the suggestion on a formal connection between our theoretical objective and practical training. We appreciate your recognition of our method design and the contributions of our work, and your positive feedback on our experimental results. Our up-to-date best result (we fixed the data loading of D&R after the submission), D&R at a budget of 512 tokens (58.5%), ranks third on the RoboMME leaderboard. D&R with a memory budget of only 64 tokens (54.8%) is competitive with it. This connects back to our theory and our goal, that the key to solving the memory problem is finding the key historical fact that the current observation lacks, and it shows that even a very limited memory budget is enough to perform well. We have updated our manuscript and provided a more detailed theoretical analysis and a more thorough ablation, which we believe can better help readers understand the core contribution of our work. We hope our clarifications can convince you to reconsider the score. Please let us know if further clarification is required.

R1 to W1 and Q1: a connection between conditional mutual information maximization and the actual flow-matching training objective.

Thank you for the comments. We added Appendix A.3, "From action likelihood to the flow-matching loss" (pages 18-19, excerpt), which gives the full derivation. In short, the connection is

I(at;mt∣ot)=∫01τ1−τ[R∗(ot,τ)−R∗((ot,mt),τ)]dτ,I(a_t;m_t\mid o_t)=\int_0^1\frac{\tau}{1-\tau}\left[R^{\ast}(o_t,\tau)-R^{\ast}((o_t,m_t),\tau)\right]d\tau,

where R∗(S,τ)=inf⁡fE[∥v−f(xτ,S,τ)∥22]R^{\ast}(S,\tau)=\inf_f\mathbb E\left[\left\Vert v-f(x_\tau,S,\tau)\right\Vert_2^2\right] is the Bayes-optimal velocity-prediction risk at flow time τ\tau given the conditioning information SS, with xτ=τat+(1−τ)ϵx_\tau=\tau a_t+(1-\tau)\epsilon and v=at−ϵv=a_t-\epsilon. A memory that lowers R∗((ot,mt),τ)R^{\ast}((o_t,m_t),\tau) at every flow time therefore carries more information. For the trained velocity network vθv_\theta and selector ψ\psi, the risk Rθ,ψ(τ)=E[∥v−vθ(xτ,(ot,mt),τ)∥22]\mathcal R_{\theta,\psi}(\tau)=\mathbb E\left[\left\Vert v-v_\theta(x_\tau,(o_t,m_t),\tau)\right\Vert_2^2\right] is never below the Bayes-optimal risk, so for any τ1∈(0,1)\tau_1\in(0,1)

I(at;mt∣ot)≥∫0τ1τ1−τ[R∗(ot,τ)−Rθ,ψ(τ)]dτ.I(a_t;m_t\mid o_t) \ge \int_0^{\tau_1}\frac{\tau}{1-\tau}\left[R^{\ast}(o_t,\tau)-\mathcal R_{\theta,\psi}(\tau)\right]d\tau .

This is the flow-matching counterpart of the variational lower bound in Eq. (7) (Corollary 1 in Appendix A.3). Lowering the flow-matching risk over the policy and the memory raises this bound, with no auxiliary memory loss or hand-designed rule. The loss of π0.5\pi_{0.5} integrates the same risk Rθ,ψ(τ)\mathcal R_{\theta,\psi}(\tau). The only difference is the weight over flow times, which is the flow-time distribution of π0.5\pi_{0.5} instead of τ/(1−τ)\tau/(1-\tau). So the actual training objective is a practical surrogate of this bound.

AuthorsOpenReview ↗

Response to Reviewer xY4U (2/4)

R2 to W2: the connection between the theoretical objective (what the objective is) and D&R (how we optimise it in practice).

The theoretical objective itself applies to any memory trained through action supervision (as detailed in the related work). What is specific to D&R is how the objective is optimised: D&R formulates it as a budgeted discrete optimisation problem, selecting original history tokens (for token fidelity) under a budget, over a history whose length grows with the episode. Empirical evidence also shows that this buys back the spatial detail that latent memories optimised through action supervision lose. We believe that filling the gap between fixed frame-selection heuristics and learned latent compression is meaningful for this line of research.

More specifically, the recursion is how we solve this discrete problem over a growing history with a selector of fixed input size. Whether the recursion helps in practice is an empirical question, and we test it in the ablation below. With the same candidate pool and budget, the recursive selector reaches 54.82 against 42.36 for the flat selector.

To help readers better understand our motivation, we refined the opening of the method section: "This section describes how we maximise the conditional mutual information I(at;mt∣ot)I(a_t;m_t\mid o_t) in practice. We optimise the memory mt=M(ht)m_t=\mathcal M(h_t) over the original history tokens, which turns it into a discrete optimisation problem, namely the selection of a subset of the history hth_t. We estimate the mapping M\mathcal M with a parametric family Mψ\mathcal M_\psi built from a selector with parameters ψ\psi. However, the history grows step by step, whereas a parametric selector has a fixed input size and should stay lightweight. Section 5.1 casts memory as budgeted discrete selection and introduces D&R, which composes the selection over a history of any length recursively from a fixed-size selector; Section 5.2 describes the selector and its training."

Optimality. We agree that the analysis does not establish that the recursion is optimal, and we do not claim it. We also agree that under a fixed token budget, zero residual information (the case where the memory is policy-sufficient, or, intuitively, where it keeps everything in the history that the action depends on) may not be reached. Our objective does not need zero residual. It only asks for the smallest residual that KK tokens can reach, because maximising the information the memory keeps is the same as minimising the residual (Eq. (3)). For a parametric method such as D&R, training moves towards this optimum, i.e., towards maximising the conditional mutual information.

R3 to Q2: ablation of recursion.

We thank the reviewer for the suggestion and ran the ablations. To separate the two factors, we also ablate parameter sharing. All rows use the same 512 candidate tokens, the same 64 output tokens and the same positional information.

SelectorContrastOverall success rate (%)
D&R (learned)
flatReference.42.36
recursive (w/o parameter sharing)Parameter sharing (vs. recursive).35.06
recursiveRecursion (vs. flat).54.82

The table separates recursion and parameter sharing, and gives the following key observations:

  1. Recursion. With the same pool and budget, the recursive selector reaches 54.82 against 42.36 for the flat selector (+12.5). This gain needs the shared selector. With a separate selector for each level, the tree reaches 35.06, below the flat selector. Each level of the tree re-scores only the tokens kept by the level below, so the least important tokens, such as background, are dropped at the leaves. The recursive structure also reduces the cost. The flat selector attends over all 512 tokens at once (5122≈2.6×105512^2\approx2.6\times10^5 token pairs per layer), whereas the tree makes seven calls on 128 tokens (7×1282≈1.1×1057\times128^2\approx1.1\times10^5). The cost of the flat selector grows quadratically with the pool, and that of the tree linearly.
  2. Parameter sharing. One selector shared by all nodes of the tree has a third of the parameters of a separate selector per level (25.2M against 75.6M), and it reaches a higher success rate (54.82 against 35.06).

The complete ablation, covering recursion, parameter sharing and the fixed selection rules, is in the newly added Appendix C.2 "Ablation: token-selection rule" (pages 24-26, excerpt).

AuthorsOpenReview ↗

Response to Reviewer xY4U (3/4)

R4 to W4: the recursive formulation handles increasingly long inputs, with a growing tree or a candidate pool.

Thank you for your suggestion. We added an experiment to test the scalability of D&R, that is, whether the recursive formulation can handle increasingly long inputs.

We added Appendix B, "Details of Divide-and-Remember" (pages 20-22, excerpt), to detail and illustrate how D&R builds the recursive tree as the history grows, with either a growing tree or a fixed tree.

(1) Growing tree. The growing history streams into the D&R selector and the recursive tree grows naturally with it. To further bound the compute, the history can be sub-sampled with a stride before it enters the tree. Only every ss-th frame of the history enters the tree as it streams in.

(2) Fixed tree (the original main results). As the history grows, we sub-sample it into a pool of fixed size, so that we only maintain a tree of fixed size with a constant number of leaves. Specifically, a fixed number of frames is sampled uniformly from the history observed so far, which gives a candidate pool of fixed size ∣H∣\vert\mathcal{H}\vert. The pool is re-drawn at every step, so the spacing between its frames widens as the history grows, while the cost of the selector stays independent of the history length.

Sub-sampling the history reduces redundancy and keeps the compute light. If the history is sub-sampled too coarsely, however, important but brief events may be dropped before the learned selector can examine them. So we designed experiments to test how densely the history needs to be sampled, at a memory budget of 64 tokens.

Success rate (%): columns Counting to AVG; Cost at deployment: columns Stored tokens to Added compute (TFLOPs).

Sub-samplingCountingPermanenceReferenceImitationAVGStored tokensPeak storage (MB)Added compute (TFLOPs)
Pool, ∣H∣=128\vert\mathcal{H}\vert=12858.4421.3326.0034.5635.0816T16T1610.03
Pool, ∣H∣=256\vert\mathcal{H}\vert=25659.1725.1724.5638.6736.8916T16T1610.05
Pool, ∣H∣=512\vert\mathcal{H}\vert=51275.4423.1754.6766.0054.8216T16T1610.11
Pool, ∣H∣=1024\vert\mathcal{H}\vert=102473.1124.9451.6759.7252.3616T16T1610.17
Stride 6, growing tree73.1723.8349.8353.4450.07min⁡(16T/6,4,096)\min(16T/6, 4{,}096)170.51
Stride 2, growing tree72.6722.6753.0058.3351.67min⁡(16T/2,16,384)\min(16T/2, 16{,}384)671.90

Stored tokens (2,048-dimensional, 16-bit each) is the most a variant needs after TT observed frames, and peak storage is its maximum over the longest possible evaluation episode (T=2,456T=2{,}456, video included). Added compute is per query and relative to π0.5\pi_{0.5} without memory (2.92 TFLOPs per query). It is independent of TT and excludes the encoding of new frames (3.54 TFLOPs per query for every memory method).

The key observations are,

  1. Temporal coverage matters for the selector to work well. For both the candidate pool and the stride, covering more frames (pool size from 128 to 512 tokens, and stride from 6 to 2) generally gives a higher success rate.
  2. The memory needs to keep the right historical evidence, instead of a very large budget. With a memory budget of only 64 tokens, D&R reaches a success rate of 54.82, against 32.22 for the best baseline (HAMLET) at the same budget. It is also competitive with D&R at a memory budget of 512 tokens (58.54, the third place on the RoboMME leaderboard). This tells us that the most important thing is to spend the context in the right place.
  3. Storage. The candidate pool is re-drawn from the whole history at every step, so it stores the tokens of every observed frame, and its storage grows linearly with the episode length (16T16T tokens, 161 MB for the longest episode). The growing tree stores only the frames that enter it, up to a fixed capacity, so its storage is bounded (17 MB at stride 6 and 67 MB at stride 2).
  4. Compute. The added compute does not grow with the episode length for any variant. The candidate pool adds 0.03 to 0.17 TFLOPs to the 2.92 TFLOPs of π0.5\pi_{0.5}, and the growing tree adds 0.51 TFLOPs at stride 6 and 1.90 TFLOPs at stride 2. The pool of 512 tokens gives the highest success rate at a small compute cost, so we use it for the main results.

The fixed tree caches nothing. It re-draws the pool from the stored frame tokens and recomputes the whole tree at every query. The growing tree never re-samples, and it keeps a node output across queries only when its stream exceeds the capacity.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta Response

Thank you very much for the reviewers' feedback. We appreciate the time and effort that the reviewers dedicated to our manuscript, and we are grateful for the insightful comments and the valuable suggestions for improving our paper. We have addressed each of the reviewers' comments individually and updated our manuscript accordingly. The main updates are as follows:

  1. Our up-to-date experiments show that D&R reaches an average success rate of 54.8% over the four suites, the highest among all compared methods, with a memory budget of only 64 tokens. This is competitive with D&R at a budget of 512 tokens (58.5%, the third place on the RoboMME leaderboard). This goes back to our theoretical motivation that the key to solving the memory problem is finding the key historical fact that the current observation lacks, and it shows that even a very limited memory budget is enough to perform well. The evaluation data (the average and standard deviation of all methods over the 80k, 90k and 100k checkpoints and seeds 0/7/42) can be found in our Anonymous GitHub repository (https://anonymous.4open.science/r/Divide-and-Remember-6363/). The code and checkpoints will also be released after the double-blind review.
  2. We added Appendix A.3, "From action likelihood to the flow-matching loss" (pages 18-19, excerpt), and refined the opening of the method section, to address the concern about the connection between conditional mutual information maximization, the actual flow-matching training objective, and the recursive memory design. Overall, the connection has three steps: what the theoretical objective is (the conditional mutual information II), how a method optimises this objective (D&R formulates it as a budgeted discrete optimisation problem over a growing history), and how the optimisation is carried out in practice (a recursive method in which a fixed-size parametric selector performs the discrete optimisation over a history of any length).
  3. We added Appendix C.2, "Ablation: token-selection rule" (pages 24-26, excerpt), to address the concern about the ablation of recursion and parameter sharing, and of different token pruning rules.
  4. We added Appendix C.4.3, "Ablation: historical token or latent token" (pages 31-32, excerpt), to isolate the benefit of raw visual tokens over latent memory. Only fusing the selected visual tokens into a latent representation of the same size lowers the average success rate from 54.82 to 28.93.
  5. We added Appendix B, "Details of Divide-and-Remember" (pages 20-22, excerpt), to detail and illustrate how D&R builds the recursive tree as the history grows, and to show that the recursive formulation handles increasingly long inputs, with a growing tree or a candidate pool. It also reports the storage and compute at deployment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.