acceptodds
Under review as a conference paper at ICLR 2027

SegMem: Segment-wise Procedural Memory with Future-aware Gating for Long-Horizon Robotic Manipulation

Abstract

Long-horizon robotic manipulation requires conditioning on procedural history. However, naively accumulating all past observations causes inference latency to grow unboundedly, which is intolerable in the real world. This necessitates organizing manipulation memory under a strictly controlled context budget. Prior work typically addresses this by either retaining sparse visual frames, risking collapse when a critical evidence is omitted, or repeatedly compressing the full interaction into a global latent state, where high-frequency read or write cycles induce information interference. These trade-offs highlight as a central design problem. Our pilot study reveals that segmented recurrence mitigates global state interference, yet its effectiveness remains sensitive to the chosen granularity. To this end, we propose SegMem, a segment-wise memory architecture that recurrently compresses variable-length observation subsequences into fixed-size segment slots. Crucially, to overcome the sensitivity of manual granularity, SegMem employs a future-aware gate that learns when to close an active segment based on its anticipated utility for subsequent action prediction. Under a fixed budget, SegMem significantly outperforms prior methods across both simulation benchmarks and real-world experiments. Further analyses demonstrate that the learned gate surpasses alternative segmentation strategies and remains robust under temporal shifts, establishing adaptive temporal organization as a key principle for memory-augmented robotic manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Memory, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pE7hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SegMem studies how to organize historical observations for memory-augmented robotic manipulation under a bounded policy-context budget. It compresses observations recurrently within temporal segments and preserves completed segment states without further modification. A causal boundary gate decides whether to continue the active segment or initialize a new one. The gate is supervised using differences in future action-prediction losses between paired split and merge continuations, followed by policy adaptation to the learned segmentation.

The evaluation covers six RoboMME tasks, LIBERO-Mem tasks T1–T8, and two real-robot tasks, with separate models trained for individual tasks. The reported average success rates are 26.3% on RoboMME, 33.1% on LIBERO-Mem, and 80% on the real-robot tasks. Additional experiments examine segmentation strategies, policy adaptation, within-segment aggregation, and supervision horizons.

Critical strengths
  1. Learning temporal boundaries from downstream action utility is a meaningful contribution.

The strongest contribution is the combination of independently archived segment states with an action-supervised boundary learner. This connects memory organization directly to what the policy subsequently needs, rather than relying exclusively on visual change or manually chosen intervals. The paired continuations share observations, action targets, and flow-matching randomness, which helps isolate the effect of the candidate boundary. Importantly, future information constructs training targets; the deployed gate uses only causally available inputs.

The contribution is more specific than introducing temporal granularity itself: prior work such as ReMem-VLA already distinguishes frame-level and chunk-level recurrence. SegMem’s valuable distinction is learning boundaries from downstream utility while protecting completed segment representations from subsequent updates.

  1. The experiments provide evidence for the proposed organization.

The paper evaluates genuinely history-dependent tasks and includes component controls beyond comparisons with memory-free policies. In Table 2(a), learned boundaries with policy adaptation achieve 37.3% average success across three tasks, compared with 22.7% for either random boundaries with adaptation or learned boundaries without adaptation. Figure 5 additionally compares against segment-count-matched random and visual-change strategies. These results support the usefulness of boundary placement and the need to adapt the policy to its resulting memory distribution. The real-world Bread Lift results also suggest a practical benefit for repeated-process tracking: SegMem succeeds in 12/20 trials versus 8/20 for PSC.

Critical weaknesses
  1. The comparisons do not yet cleanly establish how much improvement comes from learned segmentation rather than training and implementation differences.

The fixed-segmentation comparison is particularly important because it tests the paper’s central contribution. Appendix A.1 specifies 10,000 optimization updates for the pilot configurations. Figure 5 states that its fixed baselines follow this pilot protocol and reproduces their reported values. In contrast, the complete SegMem pipeline uses 15,000 policy-training updates followed by 10,000 adaptation updates, in addition to gate training. The manuscript does not establish that the fixed alternatives receive comparable training and adaptation. Consequently, their performance gap cannot yet be attributed confidently to adaptive boundary placement. Table 2(a) provides useful evidence against random segmentation, but it does not resolve the comparison with well-tuned fixed segmentation under matched training.

The external comparisons have related limitations. Appendix A.3.2 states that EventVLA and MemoryVLA are adapted to Qwen-OFT, whereas SegMem uses π₀.₅; these results therefore cannot isolate memory-mechanism quality. The same-backbone PSC and Global Recurrent comparisons are more informative, but their simulation training schedules and tuning procedures are insufficiently documented. Furthermore, the comparison includes FrameSamp with context integration while omitting the FrameSamp+Modulator variant identified as strongest overall in the original RoboMME study.

  1. The uncertainty and scope of the empirical evidence do not support the breadth of the superiority and robustness claims.

The reported 8.1-point overall advantage is arithmetically consistent with averaging the two benchmark means and comparing against Global Recurrent. However, the margins over the strongest baseline within each benchmark are substantially smaller: 26.3% versus 24.0% on RoboMME, corresponding to 79 versus 72 successes across 300 episodes; and 33.1% versus 31.3% on LIBERO-Mem, corresponding to 53 versus 50 successes across 160 episodes. No uncertainty across independent training seeds is reported for these main results. These observations do not demonstrate that the improvement is absent, but they leave its reproducibility unresolved.

The real-world advantage similarly comprises four additional Bread Lift successes and one additional Cup Transfer success. SegMem still fails all four five-cycle Bread Lift trials. Figure 6’s temporal-shift evidence is a single replay with a 16-step inserted wait, rather than an aggregate closed-loop robustness evaluation. These findings support promising behavior in selected settings, but not a broad claim of reliable long-horizon robustness.

The evaluation scope also requires correction or version clarification: the manuscript repeatedly describes T1–T8 as “all eight” LIBERO-Mem tasks, whereas the cited benchmark paper and official project describe ten tasks, including T9–T10 for object occlusion. The reason for this subset coverage should be stated accurately.

  1. Minor comments: in lines 406 and 408, it should be n∈1,2,3,4,5n\in\\{1,2,3,4,5\\} and n∈1,2,3n\in\\{1,2,3\\}, where the brackets are missing.
Questions for authors
  1. Were the fixed, random, visual-change, and learned segmentation variants trained with the same policy-update budget and comparable adaptation stages? Please specify these schedules and the corresponding simulation schedules for PSC and Global Recurrent. In particular, does Figure 5 compare the 10,000-update pilot models with the 25,000-update SegMem pipeline?

  2. Why are T9–T10 excluded from the claimed complete evaluation?

  3. When more than 16 segments are created, are older states discarded from storage or merely excluded from policy input? Do the counterfactual branches apply the same capacity constraint, including any eviction?

  4. Under matched training, adaptation, and memory budgets, does the learned gate retain its advantage over validation-tuned fixed segmentation and count-matched alternatives?

7 comments

We thank you for recognizing the value of learning boundaries from action utility and our supporting experimental evidence. We address your concerns below:

Q1: When more than 16 segments are created, are older states discarded from storage or merely excluded from policy input? Do the counterfactual branches apply the same capacity constraint, including any eviction?

A1: We thank the reviewer for this question. Older states are discarded from storage rather than merely excluded from the policy input. In our implementation, the memory retains at most 16 segment states (B/m = 512/32), including the active segment. When a Split would create a 17th segment, the oldest archived segment is permanently evicted in a first-in, first-out manner, so that the memory never exceeds the token budget. The counterfactual branches follow exactly the same capacity limit and eviction rule. For example, if the memory already contains 16 segments, the Split branch evicts the oldest archived segment before initializing a new one. Both branches are therefore compared under the same memory constraint as at deployment.

In practice, eviction is rarely triggered. The table below reports the average number of segments per episode and the fraction of episodes in which eviction occurs, both on the demonstrations used for policy adaptation and during closed-loop evaluation.

TaskAvg. #segments (demo)Eviction rate (demo)Avg. #segments (closed-loop)Eviction rate (closed-loop)
PickXtimes7.511%7.901%
PickHighlight7.130%8.803%
BinFill13.103%14.407%
SwingXtimes13.394%14.805%
RouteStick6.230%7.100%
PatternLock4.090%5.301%

Across tasks, the learned gate produces 4–15 segments per episode on average, and eviction occurs in at most 7% of closed-loop episodes. The eviction rule therefore affects only a small fraction of episodes in our experiments. Nevertheless, longer tasks may exceed the capacity more frequently. Exploring adaptive retention strategies, such as consolidating older segments instead of discarding them or making the gate aware of the remaining capacity, is an important direction for future work.

We have clarified the capacity constraint and eviction rule in Appendix A.2 and added these statistics to the revised manuscript.

Q2: Were the fixed, random, visual-change and learned segmentation variants trained with the same policy-update budget and comparable adaptation stages, and what are the corresponding simulation schedules for PSC and Global Recurrent? Under matched training, adaptation and memory budgets, does the learned gate retain its advantage over validation-tuned fixed segmentation and count-matched alternatives?

A2: We thank the reviewer for this careful observation. We agree that the original Fig. 5 reused the 10k-update pilot results for the fixed-segmentation variants, so their training budget did not match that of SegMem. To address this, we retrain all alternative segmentation variants with the same total policy-update budget as SegMem, i.e., 25k updates, corresponding to its 15k pretraining and 10k adaptation updates. Specifically, all variants start from the same Stage 1 checkpoint and are adapted for 10k updates under their own segmentations, mirroring the two-stage pipeline of SegMem. All variants use the same 512-token memory budget. As in Fig. 5, Random and Visual Change are count-matched to the learned gate. The only additional component of SegMem is gate training, which involves no policy updates and accounts for less than 0.4% of the total GPU-hours (Appendix B.3).

SegmentationPickXtimesPickHighlight
Fixed (8×64)2010
Fixed (16×32)2416
Fixed (32×16)3212
Random (count-matched)3014
Visual Change (count-matched)3218
Learned gate (Ours)3626

The learned gate retains its advantage under matched budgets. SegMem outperforms the strongest alternative by 4 points on PickXtimes and 8 points on PickHighlight. Even if the segment length is selected for each task based on test performance, which upper-bounds validation-based tuning, fixed segmentation reaches only 32% and 16%, respectively. The count-matched Random and Visual Change variants also remain below the learned gate. This indicates that the gain stems from where the boundaries are placed rather than from the number of segments.

Training schedules of PSC and Global Recurrent. Both baselines follow their RoboMME implementations and are trained for 10k updates, by which point we found their performance had already converged. To verify this, we further train both baselines for 25k updates. Global Recurrent achieves 20% on PickXtimes and 14% on PickHighlight, and PSC achieves 32% and 22%, respectively. These results are within 2 points of their reported results (18.7% and 12.0% for Global Recurrent, and 33.3% and 20.0% for PSC) and remain well below SegMem.

We will update Fig. 5 with these matched results in the final version.

Q3: EventVLA and MemoryVLA are adapted to Qwen-OFT whereas SegMem uses π0.5, so these comparisons cannot isolate the quality of the memory mechanism itself. Please provide comparisons with these baselines on the same π0.5 backbone, and include the FrameSamp+Modulator variant.

A3: We thank the reviewer for this important suggestion. We agree that a fair comparison of memory mechanisms requires a shared backbone. We have re-implemented EventVLA and MemoryVLA on the same π0.5 backbone used by SegMem. We additionally evaluate the FrameSamp+Modulator variant, denoted Perceptual Sampling (Modulator, PSM), following the implementation in RoboMME. All results are reported as mean ± standard deviation over three seeds.

RoboMME (%)

MethodPickXtimesPickHighlightBinFillSwingXtimesRouteStickPatternLockAvg.
EventVLA18.0 ± 4.036.7 ± 6.12.0 ± 2.04.0 ± 3.50.0 ± 0.08.0 ± 2.011.4
MemoryVLA12.0 ± 3.511.3 ± 1.22.7 ± 3.113.3 ± 5.05.3 ± 3.18.0 ± 3.58.8
PSC (Context)33.3 ± 3.120.0 ± 5.315.3 ± 3.136.0 ± 7.28.0 ± 2.015.3 ± 4.221.3
PSM (Modulator)35.3 ± 3.124.7 ± 4.215.3 ± 3.144.0 ± 5.310.0 ± 2.014.0 ± 2.023.9
SegMem (Ours)36.0 ± 2.026.0 ± 5.328.7 ± 5.051.3 ± 2.39.3 ± 1.213.3 ± 2.327.4

LIBERO-Mem (%)

MethodT1T2T3T4T5T6T7T8Avg.
EventVLA65.0 ± 8.715.0 ± 5.040.0 ± 8.715.0 ± 5.00.0 ± 0.05.0 ± 5.020.0 ± 8.70.0 ± 0.020.0
MemoryVLA15.0 ± 8.75.0 ± 5.010.0 ± 5.05.0 ± 5.05.0 ± 5.05.0 ± 5.05.0 ± 5.00.0 ± 0.06.3
PSC (Context)5.0 ± 0.03.3 ± 2.941.7 ± 10.455.0 ± 5.00.0 ± 0.03.3 ± 5.820.0 ± 5.018.3 ± 7.618.3
PSM (Modulator)11.7 ± 7.610.0 ± 5.043.3 ± 12.640.0 ± 5.03.3 ± 2.910.0 ± 5.013.3 ± 2.920.0 ± 5.019.0
SegMem (Ours)65.0 ± 8.755.0 ± 0.045.0 ± 25.038.3 ± 2.911.7 ± 2.916.7 ± 5.811.7 ± 5.820.0 ± 5.032.9

SegMem remains the best under the same backbone. With all memory methods built on π0.5, SegMem achieves the highest average success rate on both benchmarks, outperforming EventVLA and MemoryVLA by large margins. This confirms that the advantage of SegMem stems from its memory organization rather than from the choice of backbone.

Comparison with FrameSamp+Modulator. In our main comparison, all π0.5-based memory methods provide memory to the policy as context. We adopted this design to isolate the effect of memory organization from that of the integration mechanism, so that the differences among methods reflect how history is organized rather than how it is injected into the policy. Modulator-based integration improves Perceptual Sampling on both benchmarks (23.9 vs. 21.3 on RoboMME and 19.0 vs. 18.3 on LIBERO-Mem). Nevertheless, SegMem still outperforms PSM by 3.5 points on RoboMME and 13.9 points on LIBERO-Mem, with particularly large gains on BinFill (28.7 vs. 15.3) and SwingXtimes (51.3 vs. 44.0). Since the integration mechanism is orthogonal to memory organization, SegMem could also benefit from modulator-based integration, which we leave for future work.

We have updated the backbone description in Appendix A.3.2 and the corresponding results in Tables 1, 11, and 12 of the revised manuscript.

Reviewer_SDBqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses memory modeling in robotic manipulation by proposing a Segment-wise Recurrent Memory mechanism that compresses historical observations into separate recurrent memory segments. A Future-aware Gate is trained using future action prediction losses to adaptively determine segment boundaries. The method adopts a multi-stage training pipeline and is evaluated on RoboMME, LIBERO-Mem, and two real-world tasks.

Critical strengths
  1. The motivation is clear. The proposed segment-wise recurrent memory addresses the limitations of sparse visual memory, which may miss critical information, and global recurrent memory, which is prone to information interference.
  2. The Future-aware Gate learns segment boundaries using supervision derived from future action prediction losses. This design offers some novelty, as it does not require manually designed segmentation rules or annotated task stages.
Critical weaknesses
  1. My main concern is the design and effectiveness of the Future-aware Gate, particularly regarding three issues:
  • Mismatch between training and online inference. During training, the gate evaluates a single Merge/Split decision, followed by merge-only updates. However, during online inference, the gate makes continuous segmentation decisions. Moreover, the gate is supervised using a frozen VLA, which is subsequently updated in Stage 3. Therefore, the Split Advantage estimated during training may not accurately reflect the benefits of segmentation under the final policy. The gate lacks an online adaptation mechanism to adjust its decisions based on execution feedback.
  • Action loss does not necessarily reflect task success. The gate evaluates segmentation decisions using future Flow Matching Loss, which measures velocity-field prediction errors under different noise levels rather than the quality of generated actions or long-horizon task success. In particular, small differences in action loss between segmentation choices may not reliably indicate differences in actual task performance.
  • Lack of cross-task generalization evaluation. The gate is trained separately for each task, with task-specific threshold calibration. No cross-task transfer experiments are provided to demonstrate that the gate learns a generalizable memory segmentation strategy.
  1. Some important analyses are missing:
  • Relationship between gate decision quality and task success. The paper should report the Precision, Recall, F1 score, and False Positive Rate of the gate's segmentation decisions, and analyze how these metrics correlate with final task success rates.
  • Inference efficiency. Quantitative comparisons of inference latency and GPU memory usage against the baselines are missing.
Questions for authors
  1. Does the gate suffer from a training–inference mismatch, given that it is trained offline but makes sequential decisions online without an online adaptation mechanism?
  2. Can the authors analyze the correlation between Future Action Loss Difference and actual closed-loop task success? In particular, can small differences in Flow Matching Loss reliably indicate which segmentation decision is better?
  3. Have the authors explored sharing or transferring the gate across tasks, such as training on a subset of tasks and directly evaluating on unseen tasks?
  4. Can the authors provide curves relating gate decision quality to task success rates, as well as quantitative comparisons of inference latency and GPU memory usage against other memory methods?
6 comments

We thank you for recognizing our clear motivation and the novelty of the Future-aware Gate. We address your concerns below:

Q1: The gate is supervised by single-step counterfactuals under a frozen VLA, yet at inference it makes sequential boundary decisions for a policy that is subsequently adapted, so the estimated split advantage may not reflect the true benefit of segmentation under the final policy. Does the gate therefore suffer from a training–inference mismatch, particularly since it cannot adapt online to execution feedback?

A1: We thank the reviewer for this insightful comment and address the three aspects below.

Sequential decisions during gate training. We clarify that the merge-only continuation is used only within each paired counterfactual evaluation, i.e., to compare the Merge and Split branches up to the farthest query step. It does not mean that each training trajectory contains only a single boundary decision, since the replay itself makes a boundary decision at every subsequent candidate. Moreover, our two-round gate training is designed to mitigate this mismatch. In the second round, the first-round gate makes sequential boundary decisions along the replayed demonstrations, and the counterfactual supervision is recomputed at the resulting memory states. The final gate is therefore trained on memory states induced by its own sequential decisions, similar in spirit to DAgger [Ross et al., 2011].

Frozen VLA during gate training. The pretrained VLA serves as a fixed evaluator, providing the gate with a stationary supervision signal, and Stage 3 then adapts the policy to the memory organization produced by the learned gate. Empirically, the gate learned under the frozen VLA remains beneficial under the final policy. With policy adaptation, learned boundaries achieve 37.3% compared with 22.7% for random boundaries (Table 2, D vs. B). We acknowledge that the split advantage is not re-estimated under the adapted policy, and alternating refinement of the gate and policy is a promising extension.

Online adaptation. Although the gate parameters are fixed at deployment, its decisions are made online and conditioned on the current observations, the accumulated memory, and the segment age. As shown in Fig. 6, the gate postpones its split when a waiting period is inserted, indicating that it responds to the actual execution rather than following a fixed schedule. Updating the gate from execution feedback is considerably more challenging. During execution, only the outcome of the chosen decision is observed, so the counterfactual comparison is unavailable, and learning would instead have to rely on sparse task-level rewards with substantial interaction costs. We therefore adopt offline counterfactual supervision, which keeps deployment lightweight, and leave feedback-driven adaptation to future work.

We have clarified these points in Sec. 4.2 of the revised manuscript.

Q2: The gate is trained, and its splitting threshold calibrated, separately for each task. Have the authors explored sharing or transferring the gate across tasks, such as training on a subset of tasks and directly evaluating on unseen tasks?

A2: We thank the reviewer for this valuable suggestion. Yes, we have explored both sharing and transferring the gate across tasks on the six RoboMME tasks. In all settings, we keep the memory compressor and adapted policy of each target task fixed and replace only the boundary gate. The splitting threshold is recalibrated on the held-out validation trajectories of the target task, without retraining the gate or the policy. We consider two settings:

  • Single-source transfer: the gate trained on one task is directly applied to each of the other tasks.
  • Multi-source joint training: a single gate is trained with the same procedure on the pooled supervision of multiple tasks. LOO denotes a gate trained on the other five tasks, so the target task is unseen. Joint-A and Joint-B denote gates trained on {RouteStick, BinFill, SwingXtimes} and {RouteStick, PickHighlight, PatternLock}, respectively.
Target taskTask-specific gateSingle-source transfer (range)LOO (unseen)Joint-AJoint-B
PickXtimes3620–34342630
PickHighlight268–14141618$^\dagger$
BinFill2416–221620$^\dagger$22
SwingXtimes5026–462250$^\dagger$40
RouteStick10101010$^\dagger$10$^\dagger$
PatternLock1212121210$^\dagger$
Avg.26.319.118.022.321.7

†^\dagger denotes target tasks seen during gate training. For reference, Global Recurrent achieves an average of 12.0%.

The gate transfers to unseen tasks. Gates trained on other tasks retain a substantial portion of the task-specific performance, and every transfer setting outperforms Global Recurrent on average. In particular, gates trained on a subset of tasks generalize to tasks they have never seen. For example, the LOO gate reaches 34% on PickXtimes (vs. 36% with the task-specific gate), and Joint-B reaches 40% on SwingXtimes and 22% on BinFill.

Transfer depends on task relatedness. Gates transfer poorly to PickHighlight, whose boundaries are triggered by a task-specific transient cue. Moreover, including less related source tasks can degrade the jointly trained gate. For example, on SwingXtimes, the LOO gate adds PickXtimes and BinFill to the sources of Joint-B, and its success rate drops from 40% to 22%. These results suggest that the gate captures transferable boundary cues, and that selecting related source tasks is a promising direction for training a general-purpose gate.

We have added these results to Appendix B.6 of the revised manuscript.

Q3: The gate is supervised by differences in future flow-matching loss. This loss measures velocity-field regression error at sampled noise levels, not the quality of the generated actions or long-horizon task success. Can the authors analyze the correlation between the Future Action Loss Difference and actual closed-loop task success?

A3: We thank the reviewer for this insightful question. Ideally, the gate would be supervised directly by closed-loop task success. However, this would require executing both the Merge and Split branches in the environment for every candidate boundary until the end of the episode, which is prohibitively expensive and would make gate training impractical. Moreover, task success is a sparse, episode-level signal that is difficult to attribute to individual boundary decisions. We therefore use the action-generation loss, i.e., the training objective of the policy, as a dense and inexpensive proxy. To reduce the effect of stochasticity in this loss, the paired branches share the same action target, flow noise, and flow time, and Eqs. (7)–(8) temper and downweight comparisons with weak evidence.

To further examine whether this proxy is consistent with closed-loop success, we conduct analyses at both the model level and the decision level:

Action loss vs. generated actions and task success. We use the fixed-segmentation models from the pilot study (L ∈ {2, 4, 8}) and measure their FM loss on demonstrations. To directly assess the quality of generated actions, we also sample action chunks with 10-step Euler integration and compute their MSE against the demonstrated actions. Success rates are taken from Fig. 2.

TaskModelFM loss (×10⁻³) ↓Action MSE (×10⁻²) ↓Success rate (%) ↑
PickXtimesFixed L=84.102.2018
Fixed L=43.551.9024
Fixed L=23.101.5530
PickHighlightFixed L=84.202.608
Fixed L=42.901.5514
Fixed L=24.052.508
BinFillFixed L=84.203.0016
Fixed L=44.283.0816
Fixed L=24.152.9516
SwingXtimesFixed L=83.151.8550
Fixed L=43.081.8250
Fixed L=22.651.4058

On all four tasks, the ranking of the models by FM loss is identical to their ranking by generated-action MSE, indicating that a lower FM loss corresponds to more accurate generated actions. Moreover, on the three tasks where success varies across models, the model with the lowest FM loss achieves the highest success rate. On BinFill, where the losses differ only marginally, the success rates are identical. After removing per-task means, the FM loss is strongly negatively correlated with task success across the 12 models (Pearson r = −0.88).

Loss-difference supervision vs. task success. At the decision level, we examine whether following the supervision derived from the future action loss difference leads to higher success. As reported in our response to Q4, gates that agree more closely with this supervision achieve higher success rates, with a Pearson correlation between gate F1 and task success of 0.925 on PickHighlight and 0.645 on SwingXtimes.

Together, these results suggest that the future action loss difference provides a reasonable proxy for the downstream utility of boundary decisions. We have added the motivation and the analysis to Appendix B.7 of the revised manuscript.

Reviewer_WpLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a segment-wise memory architecture (SegMem) for long-horizon robotic manipulation that addresses the trade-off between historical coverage and information fidelity under a fixed memory-token budget. Instead of compressing the entire observation history into a single recurrent state, SegMem uses a Recurrent Memory Transformer (RMT) to maintain separate segment-level memories, reducing information interference. To adaptively determine segment boundaries, the authors introduce a future-aware gating mechanism trained by comparing the future action-prediction losses of merging and splitting memory segments. Experiments on RoboMME, LIBERO-Mem, and real-world manipulation tasks demonstrate that SegMem outperforms existing memory-augmented baselines, particularly on tasks requiring long-term historical information.

Critical strengths
  • The trade-off between historical coverage and information fidelity is an important challenge for memory-augmented robotic manipulation, particularly in long-horizon tasks. The paper addresses a relevant problem with practical implications for deploying VLA policies under limited context budgets.
  • Adapting RMT to maintain separate segment-wise memories is a promising approach to mitigating information interference in global recurrent memory. The pilot study provides useful empirical evidence supporting the benefits of segmented memory and motivates the proposed adaptive segmentation mechanism.
Critical weaknesses
  • The description of the RMT-based segmented memory in Section 3.2 is rather dense, making it difficult for readers unfamiliar with RMT to understand key concepts such as segment length, memory allocation, and recurrent segment updates. Providing additional illustrations or a step-by-step example, potentially in the appendix, would improve readability and help readers better understand the motivation and technical contributions of the proposed method.

  • The experiments focus primarily on tasks that explicitly require historical information. However, it is also important to evaluate the proposed method on general-purpose manipulation benchmarks, including tasks where memory is less critical or unnecessary. Such experiments would help establish whether the additional memory mechanism introduces any performance degradation or other trade-offs in tasks that can be solved effectively using reactive policies.

  • Complex and potentially computationally expensive training pipeline. The proposed method requires a multi-stage training procedure involving memory pretraining, counterfactual evaluation of Merge/Split decisions, gate training, and subsequent policy adaptation. In particular, generating supervision signals for the future-aware gate requires constructing two counterfactual branches and evaluating their action-prediction losses at multiple future time steps. This introduces considerable implementation complexity and potentially substantial computational overhead. However, the paper does not provide a detailed analysis of the additional training cost, making it difficult to assess the practical scalability and efficiency of the approach.

  • Limited evaluation of multi-task generalization. As acknowledged by the authors, each model is trained and evaluated independently on a single task due to computational constraints. Consequently, it remains unclear whether the proposed future-aware gating mechanism can be effectively trained across diverse manipulation tasks or generalize to unseen tasks. Given the additional complexity of the training procedure, evaluating its scalability to multi-task settings would be valuable, particularly for general-purpose VLA policies.

  • Sensitivity to manually selected future-query horizons. The future-aware gate relies on comparing the action-prediction losses of Merge and Split branches at a predefined set of future time steps. For example, the authors use offsets of 0, 16, 32, and 64 control steps on RoboMME. However, the motivation for selecting these particular offsets is not sufficiently explained. Since the estimated advantage of splitting depends on the selected future horizons, the resulting supervision signals may be sensitive to these choices. Although Figure 5 compares short-term, long-term, and mixed-horizon supervision, a more systematic sensitivity analysis would help establish the robustness of the proposed gate-training strategy.

Questions for authors

Questions below are relevant to the aforementioned weaknesses:

  • How does SegMem perform on standard manipulation tasks that do not require long-term historical information?
  • Could the authors quantify the additional training time, GPU resources, or computational cost introduced by the future-aware gate, particularly the counterfactual Merge/Split evaluations?
  • Can a single future-aware gate be trained jointly across multiple manipulation tasks?
  • What motivates the specific future-query offsets used for gate supervision (e.g., 0, 16, 32, and 64 control steps on RoboMME)? How sensitive is the learned gate to these choices?
5 comments

We thank you for recognizing the importance of the coverage–fidelity trade-off and the promise of segment-wise memory. We address your concerns below:

Q1: The description of the Recurrent Memory Transformer (RMT)-based segmented memory in Sec. 3.2 may be difficult to follow for readers unfamiliar with RMT, particularly regarding segment length, memory allocation, and recurrent segment updates. Additional illustrations would help clarify the motivation and technical contributions.

A1: We thank the reviewer for this valuable suggestion. We agree that the original description in Sec. 3.2 was overly compact, and we clarify the key concepts below.

Standard RMT. A standard RMT maintains a single global memory state of BB tokens. At each memory update, the recurrent Transformer TϕT_\phi takes the current memory state concatenated with the current observation tokens as input. It outputs the updated memory state together with the contextualized observation tokens (Eq. (2)). Because this single state is rewritten at every update, information from earlier task phases is repeatedly re-compressed with later observations, which can lead to representational interference.

From standard RMT to segmented RMT. The two variants differ in how long a memory state keeps being updated. A standard RMT iterates a single state over the entire interaction history. In contrast, a segmented RMT restricts recurrent updates to the currently active segment and freezes each segment once it is completed.

Segmented RMT. The NN historical memory updates are divided into consecutive segments of length LL, where LL counts memory updates rather than environment steps. Within a segment, only the active state is updated with each incoming observation. Once the segment is complete, its state is archived and no longer updated, and a new segment is initialized from a shared learned state. The policy-facing memory concatenates all archived states with the current active state (Eq. (3)). The standard RMT thus corresponds to the special case of a single segment.

Memory allocation. All configurations share the same total budget of B=512B = 512 tokens, which is evenly divided among the segments. With N=64N = 64 memory updates in our pilot study:

ConfigurationSegment length LLNum. segmentsTokens per segmentTotal tokens
Global RMT641512512
Fixed L=8L=88864512
Fixed L=4L=441632512
Fixed L=2L=223216512

The configurations therefore differ only in how the memory is organized, not in its total capacity. A smaller LL gives finer temporal resolution but fewer tokens per segment. A larger LL gives more capacity per segment but coarser temporal organization. Since the best LL varies across tasks (Fig. 2), SegMem replaces the fixed segment length with a learned future-aware gate (Sec. 4).

Following the reviewer's suggestion, we have revised Sec. 3.2 and added a schematic illustration contrasting the standard and segmented RMT (Fig. 2). We have also added a step-by-step example of the segmented memory update in Appendix A.1. We kindly invite the reviewer to refer to the revised manuscript.

Q2: How does SegMem perform on standard manipulation tasks where long-term history is less critical? Such results would help show whether the memory mechanism introduces any trade-offs compared with reactive policies.

A2: We thank the reviewer for this insightful question. To examine whether the memory mechanism introduces trade-offs when long-term history is less critical, we evaluate SegMem in two additional settings: the standard LIBERO benchmark and the short-horizon tasks of LIBERO-Mem.

Standard manipulation tasks (LIBERO). We evaluate on three standard LIBERO suites, namely Spatial, Object, and Goal. Each suite contains 10 short-horizon tabletop tasks, which vary in spatial layout, object category, and task goal, respectively. These tasks can largely be solved from the current observation and are therefore well suited to reactive policies. We compare SegMem with the history-free π0.5\pi_{0.5} policy and PSC, all built on the same π0.5\pi_{0.5} backbone. We report the mean ±\pm std over three random seeds.

MethodSpatialObjectGoalAvg.
π0.5\pi_{0.5} (history-free)96.67 ±\pm 1.6698.92 ±\pm 0.7296.67 ±\pm 1.2397.42 ±\pm 0.73
PSC97.08 ±\pm 1.7098.50 ±\pm 1.3096.42 ±\pm 2.3197.33 ±\pm 0.25
SegMem (Ours)98.58 ±\pm 0.5898.50 ±\pm 1.1596.33 ±\pm 0.5297.81 ±\pm 0.35

SegMem achieves the highest average success rate of 97.81%. It improves over π0.5\pi_{0.5} by 1.9 points on LIBERO-Spatial, while on LIBERO-Object and LIBERO-Goal it stays within one standard deviation of the best method. These results indicate that SegMem does not degrade performance on standard manipulation tasks that can be solved reactively.

Short-horizon tasks in LIBERO-Mem. We further examine T1 (Bowl$\times$1) and T2 (Bottle$\times$1) from LIBERO-Mem, in which the robot lifts an object and returns it to its plate only once. These tasks are short-horizon and require only minimal history, namely whether the single lift-and-return operation has already been completed.

MethodT1 (Bowl$\times$1)T2 (Bottle$\times$1)Avg.
π0.5\pi_{0.5} (history-free)45.0 ±\pm 10.023.3 ±\pm 12.634.2
PSC5.0 ±\pm 0.03.3 ±\pm 2.94.2
SegMem (Ours)65.0 ±\pm 8.755.0 ±\pm 0.060.0

SegMem improves over π0.5\pi_{0.5} on both tasks. In contrast, PSC performs substantially worse than the history-free policy. This shows that naively incorporating historical observations can harm performance even on short-horizon tasks, whereas SegMem does not exhibit such a trade-off.

Discussion. We attribute this behavior to the design of SegMem. The memory tokens are provided as additional context alongside the full current observation (Eq. (1)), so the policy retains all the information needed for reactive control. The compact segment memory therefore serves as complementary context rather than a replacement for the current input. Overall, these results suggest that SegMem can act as a general-purpose memory module: it benefits history-dependent tasks without sacrificing performance on standard manipulation tasks.

Q3: What is the additional training cost of the multi-stage training pipeline? Please provide the additional training time and GPU resources introduced by the future-aware gate, particularly the counterfactual Merge/Split evaluations.

A3: We thank the reviewer for this question. All experiments are conducted on NVIDIA A800 GPUs. Stages 1 and 3 use 8 GPUs, and Stage 2 (gate learning) uses a single GPU. The per-task training cost of each stage is summarized below.

StageComponentGPUsWall-clock timeGPU-hours
Stage 1Memory pretraining (15k updates)825–26 h200–208
Stage 2Counterfactual Merge/Split evaluation (2 rounds)114–38 min0.23–0.63
Stage 2Gate training (2 rounds, 100 epochs each)12–7 min0.03–0.12
Stage 3Policy adaptation (10k updates)81.5–2 h12–16
Total212–225

The future-aware gate, including the counterfactual Merge/Split evaluation, takes only 16–45 minutes per task on a single GPU, which accounts for less than 0.4% of the total GPU-hours. The gate itself is a lightweight network with only about 0.375M parameters, and the counterfactual evaluation requires only forward passes of the frozen Stage 1 model. We have added these results to Appendix B.3 of the revised manuscript.

Q4: Since each model is trained and evaluated on a single task, it remains unclear whether the future-aware gating mechanism can be effectively trained across diverse manipulation tasks or generalize to unseen tasks. Can a single future-aware gate be trained jointly across multiple manipulation tasks?

A4: We thank the reviewer for this insightful question. To examine this, we keep the memory compressor and adapted policy of each target task fixed and replace only its boundary gate. We evaluate two settings across six RoboMME tasks: (i) single-source transfer, where the gate trained on one task is applied to each of the other tasks; and (ii) multi-source joint training, where a single gate is trained with the same procedure on the pooled supervision of multiple tasks. For joint training, LOO denotes a gate trained on the other five tasks (target unseen), and Joint-A/B denote gates trained on {RouteStick, BinFill, SwingXtimes} and {RouteStick, PickHighlight, PatternLock}, respectively.

Target taskTask-specific gateSingle-source transfer (range)LOO (unseen)Joint-AJoint-B
PickXtimes3620–34342630
PickHighlight268–14141618$^\dagger$
BinFill2416–221620$^\dagger$22
SwingXtimes5026–462250$^\dagger$40
RouteStick10101010$^\dagger$10$^\dagger$
PatternLock1212121210$^\dagger$
Avg.26.319.118.022.321.7

†^\dagger denotes target tasks seen during gate training. For reference, Global Recurrent achieves an average of 12.0%.

We observe the following key findings:

1) A single gate can be trained jointly across tasks. Jointly trained gates remain effective on their training tasks and also generalize to unseen tasks. Similarly, gates transferred from other tasks retain a substantial portion of the task-specific performance, and all transfer settings outperform Global Recurrent on average.

2) Transfer depends on task relatedness. Gates transfer poorly to PickHighlight, whose boundaries are triggered by a task-specific transient cue. Moreover, including less related source tasks can degrade the joint gate. For example, on SwingXtimes, the LOO gate adds PickXtimes and BinFill to the sources of Joint-B, and its success rate drops from 40% to 22%. These results suggest that the gate captures transferable boundary cues, and that selecting related source tasks is a promising direction for training a general-purpose gate at scale.

We have added these results to Appendix B.6 of the revised manuscript.

Q5: What motivates the choice of future-query offsets for gate supervision (e.g., {0, 16, 32, 64} control steps on RoboMME)? Please provide a sensitivity analysis of the learned gate with respect to these choices.

A5: We thank the reviewer for this question. Following prior work that supervises decision-making with predictions at multiple future time steps [1], we place the future queries at exponentially increasing offsets. This covers both the immediate and the delayed consequences of a boundary decision with only a few queries. Our analysis further confirms that this is a favorable choice. Specifically, we extend the comparison in Fig. 5 with additional configurations on PickHighlight, varying the range, spacing, and number of queries. For each configuration, we regenerate the counterfactual supervision, retrain the gate, and adapt the policy. Results are averaged over two training seeds and three inference seeds (300 episodes in total).

SettingOffsets (control steps)#QueriesSuccess rate (%)
Mixed (default){0, 16, 32, 64}4<u>26.0</u>
Short-term{0, 16, 32, 48}416.0
Long-term{0, 32, 64, 128}416.0
Shifted{0, 16, 48, 64}425.3
Sparse{0, 64}214.0
Dense{0, 16, 32, 48, 64}526.7
  • The gate does not rely on a precise query position. Moving an intermediate query from 32 to 48 (Shifted) changes performance by less than one point.
  • Balanced coverage of near and distant horizons is important. Restricting the queries to nearer (Short-term) or more distant (Long-term) horizons reduces performance by 10 points, and removing the intermediate queries (Sparse) causes the largest drop.
  • Additional queries bring only marginal gains. Adding a fifth query (Dense) improves performance by only 0.7 points, while increasing the cost of counterfactual evaluation.

Overall, the default layout offers a practical trade-off between temporal coverage and computational cost, and moderate variations of the offsets have limited effect on performance. We have added this analysis to Appendix B.4 of the revised manuscript.

[1] Dosovitskiy A, Koltun V. Learning to act by predicting the future[J]. ICLR2017.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.