- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SegMem studies how to organize historical observations for memory-augmented robotic manipulation under a bounded policy-context budget. It compresses observations recurrently within temporal segments and preserves completed segment states without further modification. A causal boundary gate decides whether to continue the active segment or initialize a new one. The gate is supervised using differences in future action-prediction losses between paired split and merge continuations, followed by policy adaptation to the learned segmentation.
The evaluation covers six RoboMME tasks, LIBERO-Mem tasks T1–T8, and two real-robot tasks, with separate models trained for individual tasks. The reported average success rates are 26.3% on RoboMME, 33.1% on LIBERO-Mem, and 80% on the real-robot tasks. Additional experiments examine segmentation strategies, policy adaptation, within-segment aggregation, and supervision horizons.
Critical strengths
- Learning temporal boundaries from downstream action utility is a meaningful contribution.
The strongest contribution is the combination of independently archived segment states with an action-supervised boundary learner. This connects memory organization directly to what the policy subsequently needs, rather than relying exclusively on visual change or manually chosen intervals. The paired continuations share observations, action targets, and flow-matching randomness, which helps isolate the effect of the candidate boundary. Importantly, future information constructs training targets; the deployed gate uses only causally available inputs.
The contribution is more specific than introducing temporal granularity itself: prior work such as ReMem-VLA already distinguishes frame-level and chunk-level recurrence. SegMem’s valuable distinction is learning boundaries from downstream utility while protecting completed segment representations from subsequent updates.
- The experiments provide evidence for the proposed organization.
The paper evaluates genuinely history-dependent tasks and includes component controls beyond comparisons with memory-free policies. In Table 2(a), learned boundaries with policy adaptation achieve 37.3% average success across three tasks, compared with 22.7% for either random boundaries with adaptation or learned boundaries without adaptation. Figure 5 additionally compares against segment-count-matched random and visual-change strategies. These results support the usefulness of boundary placement and the need to adapt the policy to its resulting memory distribution. The real-world Bread Lift results also suggest a practical benefit for repeated-process tracking: SegMem succeeds in 12/20 trials versus 8/20 for PSC.
Critical weaknesses
- The comparisons do not yet cleanly establish how much improvement comes from learned segmentation rather than training and implementation differences.
The fixed-segmentation comparison is particularly important because it tests the paper’s central contribution. Appendix A.1 specifies 10,000 optimization updates for the pilot configurations. Figure 5 states that its fixed baselines follow this pilot protocol and reproduces their reported values. In contrast, the complete SegMem pipeline uses 15,000 policy-training updates followed by 10,000 adaptation updates, in addition to gate training. The manuscript does not establish that the fixed alternatives receive comparable training and adaptation. Consequently, their performance gap cannot yet be attributed confidently to adaptive boundary placement. Table 2(a) provides useful evidence against random segmentation, but it does not resolve the comparison with well-tuned fixed segmentation under matched training.
The external comparisons have related limitations. Appendix A.3.2 states that EventVLA and MemoryVLA are adapted to Qwen-OFT, whereas SegMem uses π₀.₅; these results therefore cannot isolate memory-mechanism quality. The same-backbone PSC and Global Recurrent comparisons are more informative, but their simulation training schedules and tuning procedures are insufficiently documented. Furthermore, the comparison includes FrameSamp with context integration while omitting the FrameSamp+Modulator variant identified as strongest overall in the original RoboMME study.
- The uncertainty and scope of the empirical evidence do not support the breadth of the superiority and robustness claims.
The reported 8.1-point overall advantage is arithmetically consistent with averaging the two benchmark means and comparing against Global Recurrent. However, the margins over the strongest baseline within each benchmark are substantially smaller: 26.3% versus 24.0% on RoboMME, corresponding to 79 versus 72 successes across 300 episodes; and 33.1% versus 31.3% on LIBERO-Mem, corresponding to 53 versus 50 successes across 160 episodes. No uncertainty across independent training seeds is reported for these main results. These observations do not demonstrate that the improvement is absent, but they leave its reproducibility unresolved.
The real-world advantage similarly comprises four additional Bread Lift successes and one additional Cup Transfer success. SegMem still fails all four five-cycle Bread Lift trials. Figure 6’s temporal-shift evidence is a single replay with a 16-step inserted wait, rather than an aggregate closed-loop robustness evaluation. These findings support promising behavior in selected settings, but not a broad claim of reliable long-horizon robustness.
The evaluation scope also requires correction or version clarification: the manuscript repeatedly describes T1–T8 as “all eight” LIBERO-Mem tasks, whereas the cited benchmark paper and official project describe ten tasks, including T9–T10 for object occlusion. The reason for this subset coverage should be stated accurately.
- Minor comments: in lines 406 and 408, it should be and , where the brackets are missing.
Questions for authors
-
Were the fixed, random, visual-change, and learned segmentation variants trained with the same policy-update budget and comparable adaptation stages? Please specify these schedules and the corresponding simulation schedules for PSC and Global Recurrent. In particular, does Figure 5 compare the 10,000-update pilot models with the 25,000-update SegMem pipeline?
-
Why are T9–T10 excluded from the claimed complete evaluation?
-
When more than 16 segments are created, are older states discarded from storage or merely excluded from policy input? Do the counterfactual branches apply the same capacity constraint, including any eviction?
-
Under matched training, adaptation, and memory budgets, does the learned gate retain its advantage over validation-tuned fixed segmentation and count-matched alternatives?
7 comments
Q2: Were the fixed, random, visual-change and learned segmentation variants trained with the same policy-update budget and comparable adaptation stages, and what are the corresponding simulation schedules for PSC and Global Recurrent? Under matched training, adaptation and memory budgets, does the learned gate retain its advantage over validation-tuned fixed segmentation and count-matched alternatives?
A2: We thank the reviewer for this careful observation. We agree that the original Fig. 5 reused the 10k-update pilot results for the fixed-segmentation variants, so their training budget did not match that of SegMem. To address this, we retrain all alternative segmentation variants with the same total policy-update budget as SegMem, i.e., 25k updates, corresponding to its 15k pretraining and 10k adaptation updates. Specifically, all variants start from the same Stage 1 checkpoint and are adapted for 10k updates under their own segmentations, mirroring the two-stage pipeline of SegMem. All variants use the same 512-token memory budget. As in Fig. 5, Random and Visual Change are count-matched to the learned gate. The only additional component of SegMem is gate training, which involves no policy updates and accounts for less than 0.4% of the total GPU-hours (Appendix B.3).
| Segmentation | PickXtimes | PickHighlight |
|---|---|---|
| Fixed (8×64) | 20 | 10 |
| Fixed (16×32) | 24 | 16 |
| Fixed (32×16) | 32 | 12 |
| Random (count-matched) | 30 | 14 |
| Visual Change (count-matched) | 32 | 18 |
| Learned gate (Ours) | 36 | 26 |
The learned gate retains its advantage under matched budgets. SegMem outperforms the strongest alternative by 4 points on PickXtimes and 8 points on PickHighlight. Even if the segment length is selected for each task based on test performance, which upper-bounds validation-based tuning, fixed segmentation reaches only 32% and 16%, respectively. The count-matched Random and Visual Change variants also remain below the learned gate. This indicates that the gain stems from where the boundaries are placed rather than from the number of segments.
Training schedules of PSC and Global Recurrent. Both baselines follow their RoboMME implementations and are trained for 10k updates, by which point we found their performance had already converged. To verify this, we further train both baselines for 25k updates. Global Recurrent achieves 20% on PickXtimes and 14% on PickHighlight, and PSC achieves 32% and 22%, respectively. These results are within 2 points of their reported results (18.7% and 12.0% for Global Recurrent, and 33.3% and 20.0% for PSC) and remain well below SegMem.
We will update Fig. 5 with these matched results in the final version.
Q3: EventVLA and MemoryVLA are adapted to Qwen-OFT whereas SegMem uses π0.5, so these comparisons cannot isolate the quality of the memory mechanism itself. Please provide comparisons with these baselines on the same π0.5 backbone, and include the FrameSamp+Modulator variant.
A3: We thank the reviewer for this important suggestion. We agree that a fair comparison of memory mechanisms requires a shared backbone. We have re-implemented EventVLA and MemoryVLA on the same π0.5 backbone used by SegMem. We additionally evaluate the FrameSamp+Modulator variant, denoted Perceptual Sampling (Modulator, PSM), following the implementation in RoboMME. All results are reported as mean ± standard deviation over three seeds.
RoboMME (%)
| Method | PickXtimes | PickHighlight | BinFill | SwingXtimes | RouteStick | PatternLock | Avg. |
|---|---|---|---|---|---|---|---|
| EventVLA | 18.0 ± 4.0 | 36.7 ± 6.1 | 2.0 ± 2.0 | 4.0 ± 3.5 | 0.0 ± 0.0 | 8.0 ± 2.0 | 11.4 |
| MemoryVLA | 12.0 ± 3.5 | 11.3 ± 1.2 | 2.7 ± 3.1 | 13.3 ± 5.0 | 5.3 ± 3.1 | 8.0 ± 3.5 | 8.8 |
| PSC (Context) | 33.3 ± 3.1 | 20.0 ± 5.3 | 15.3 ± 3.1 | 36.0 ± 7.2 | 8.0 ± 2.0 | 15.3 ± 4.2 | 21.3 |
| PSM (Modulator) | 35.3 ± 3.1 | 24.7 ± 4.2 | 15.3 ± 3.1 | 44.0 ± 5.3 | 10.0 ± 2.0 | 14.0 ± 2.0 | 23.9 |
| SegMem (Ours) | 36.0 ± 2.0 | 26.0 ± 5.3 | 28.7 ± 5.0 | 51.3 ± 2.3 | 9.3 ± 1.2 | 13.3 ± 2.3 | 27.4 |
LIBERO-Mem (%)
| Method | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| EventVLA | 65.0 ± 8.7 | 15.0 ± 5.0 | 40.0 ± 8.7 | 15.0 ± 5.0 | 0.0 ± 0.0 | 5.0 ± 5.0 | 20.0 ± 8.7 | 0.0 ± 0.0 | 20.0 |
| MemoryVLA | 15.0 ± 8.7 | 5.0 ± 5.0 | 10.0 ± 5.0 | 5.0 ± 5.0 | 5.0 ± 5.0 | 5.0 ± 5.0 | 5.0 ± 5.0 | 0.0 ± 0.0 | 6.3 |
| PSC (Context) | 5.0 ± 0.0 | 3.3 ± 2.9 | 41.7 ± 10.4 | 55.0 ± 5.0 | 0.0 ± 0.0 | 3.3 ± 5.8 | 20.0 ± 5.0 | 18.3 ± 7.6 | 18.3 |
| PSM (Modulator) | 11.7 ± 7.6 | 10.0 ± 5.0 | 43.3 ± 12.6 | 40.0 ± 5.0 | 3.3 ± 2.9 | 10.0 ± 5.0 | 13.3 ± 2.9 | 20.0 ± 5.0 | 19.0 |
| SegMem (Ours) | 65.0 ± 8.7 | 55.0 ± 0.0 | 45.0 ± 25.0 | 38.3 ± 2.9 | 11.7 ± 2.9 | 16.7 ± 5.8 | 11.7 ± 5.8 | 20.0 ± 5.0 | 32.9 |
SegMem remains the best under the same backbone. With all memory methods built on π0.5, SegMem achieves the highest average success rate on both benchmarks, outperforming EventVLA and MemoryVLA by large margins. This confirms that the advantage of SegMem stems from its memory organization rather than from the choice of backbone.
Comparison with FrameSamp+Modulator. In our main comparison, all π0.5-based memory methods provide memory to the policy as context. We adopted this design to isolate the effect of memory organization from that of the integration mechanism, so that the differences among methods reflect how history is organized rather than how it is injected into the policy. Modulator-based integration improves Perceptual Sampling on both benchmarks (23.9 vs. 21.3 on RoboMME and 19.0 vs. 18.3 on LIBERO-Mem). Nevertheless, SegMem still outperforms PSM by 3.5 points on RoboMME and 13.9 points on LIBERO-Mem, with particularly large gains on BinFill (28.7 vs. 15.3) and SwingXtimes (51.3 vs. 44.0). Since the integration mechanism is orthogonal to memory organization, SegMem could also benefit from modulator-based integration, which we leave for future work.
We have updated the backbone description in Appendix A.3.2 and the corresponding results in Tables 1, 11, and 12 of the revised manuscript.
We thank you for recognizing the value of learning boundaries from action utility and our supporting experimental evidence. We address your concerns below:
Q1: When more than 16 segments are created, are older states discarded from storage or merely excluded from policy input? Do the counterfactual branches apply the same capacity constraint, including any eviction?
A1: We thank the reviewer for this question. Older states are discarded from storage rather than merely excluded from the policy input. In our implementation, the memory retains at most 16 segment states (B/m = 512/32), including the active segment. When a Split would create a 17th segment, the oldest archived segment is permanently evicted in a first-in, first-out manner, so that the memory never exceeds the token budget. The counterfactual branches follow exactly the same capacity limit and eviction rule. For example, if the memory already contains 16 segments, the Split branch evicts the oldest archived segment before initializing a new one. Both branches are therefore compared under the same memory constraint as at deployment.
In practice, eviction is rarely triggered. The table below reports the average number of segments per episode and the fraction of episodes in which eviction occurs, both on the demonstrations used for policy adaptation and during closed-loop evaluation.
Across tasks, the learned gate produces 4–15 segments per episode on average, and eviction occurs in at most 7% of closed-loop episodes. The eviction rule therefore affects only a small fraction of episodes in our experiments. Nevertheless, longer tasks may exceed the capacity more frequently. Exploring adaptive retention strategies, such as consolidating older segments instead of discarding them or making the gate aware of the remaining capacity, is an important direction for future work.
We have clarified the capacity constraint and eviction rule in Appendix A.2 and added these statistics to the revised manuscript.