- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
FlexMemWAM manages visual history through state-guided temporal segmentation and learned segment compression. Each variable-length segment shares a fixed set of memory slots, whose layer-wise K/V representations support subsequent action prediction. Experiments on RMBench and three real-world tasks demonstrate promising performance.
Critical strengths
- Well-motivated memory management. Jointly compressing consecutive observations provides a sensible way to reduce redundant memory allocation while retaining historical evidence.
- Careful causal design. Boundary confirmation, attention visibility, and cache updates are explicitly specified, supporting online execution without requiring future information.
- Effective learned compression. Segment memory is integrated into the native WAM layers and optimized through joint video and action prediction.
- Component ablations support the contributions of the initial anchor and segment memory, while simulation and real-world evaluations cover several forms of history dependence.
- The paper clearly distinguishes cache-capacity analysis from runtime evaluation and acknowledges the limitations of predefined segmentation signals.
Critical weaknesses
- Missing real-world inference latency evaluation. Cache savings alone do not establish practical efficiency. Segment writing and suffix recomputation introduce additional computation, but their effects on end-to-end latency and control responsiveness are not measured.
- Hand-crafted grouping with insufficiently established generality. Although compression is learned, segmentation and write triggering rely on predefined robot-state cues, thresholds, and length constraints. These are reasonable for the evaluated tasks, but may miss important visual changes without corresponding robot motion.
- Insufficient visualization of variable-length segments. The grouping and writing procedure is relatively complex. Actual trajectories showing state signals, boundaries, confirmation delays, and segment lengths would help establish whether the resulting groups align with meaningful interactions.
- The ablation study covers only two tasks and could more thoroughly examine adaptive segmentation. Although Fixed-4 provides a uniform-segmentation baseline, the comparison does not control for total cache capacity: using eight slots per segment yields different memory budgets when the number of segments differs. Comparing adaptive and uniform segmentation under matched total cache budgets would better isolate the benefit of boundary placement from differences in memory allocation.
Questions for authors
- Could you report real-robot end-to-end latency, including median and tail latency, with a breakdown of encoding, memory writing, suffix recomputation, and action denoising? How does this compare with per-frame compression and LingBot-VA?
- Are the same segmentation rules and hyperparameters used across all simulation and real-world tasks, or are they adjusted for individual tasks?
- How sensitive is performance to the segmentation thresholds and segment-length constraints?
- Could you visualize representative segmentations, including failure cases, and compare against random boundaries matched to the proposed method’s segment-length distribution?
- Removing the initial anchor causes substantial performance drops on the two ablated tasks. Why is the model so dependent on the anchor, and does this dependence hold across the other tasks? Could you provide further analysis and broader ablations?
2 comments
Response to Reviewer erFR
Q5. Anchor dependence across tasks
The anchor provides an initial-scene reference for both memory writing and action prediction. For example, Put Back Block requires remembering the block's location before it is moved. To test whether the dependence extends beyond the original two tasks, we added Press Button and Put Back Block:
| Task | Full model | Without anchor |
|---|---|---|
| Swap T | 76% | 16% |
| Cover Blocks | 100% | 61% |
| Press Button | 67% | 26% |
| Put Back Block | 100% | 33% |
Removing the anchor lowers success across all four tasks. These tasks include both those requiring a single critical historical state and those requiring history across multiple interactions. The dependence therefore extends beyond the two original ablations.
The anchor and segment memory provide complementary information: the initial scene and the subsequent interaction history. On Swap T and Cover Blocks, removing segment memory reduces success even when the anchor is retained. The benefit of segment memory therefore extends beyond simply preserving the initial observation.
W1. Runtime efficiency and responsiveness
We added execution-oriented timing to examine the practical effect of memory writing and suffix recomputation. On RTX 4090 with recorded FR3 inputs, software end-to-end p50/p95 is 189.73/226.91 ms for FlexMemWAM, 207.06/209.42 ms for Frame-wise, and 914.46/1117.92 ms for LingBot-VA.
Relative to Frame-wise, the median is 8.4% lower and p95 is 8.4% higher. In the separate FR3 8D stage test, encoding, memory writing, suffix recomputation, and action denoising average 22.08, 5.99, 1.35, and 130.33 ms per step, respectively. Intermittent writing reduces average per-step memory-writing cost, while writing and recomputation at segment closure add computation, resulting in lower median but higher tail latency.
W2. Generality of state-based grouping
The robot-state cues determine when a segment is closed. They do not restrict its contents to observations that trigger motion events. A sampled visual change remains accessible through the recent/open context and participates in joint compression at closure, even if it does not create a boundary itself. Long-term retention without robot motion remains untested.
To examine the value of observations within a segment, we compared joint compression with an uncompressed segment-end keyframe baseline on Battery Try. The baseline uses the same boundaries and directly inputs segment endpoints plus the initial and current observations. FlexMemWAM achieves 83% success rate, compared with 71% for the keyframe baseline. This shows that retaining endpoints alone does not recover the benefit of jointly represented segment history.
The configuration is shared within each platform: one for RMBench and another for FR3, without per-task tuning. Changing stabilization from 0.95 to 0.9 and gripper thresholds from 0.2/0.8 to 0.1/0.9 changes Swap T from 76% to 74%. Widening segment lengths from 3–8 to 2–10 changes it from 76% to 73%. Put Back Block remains at 100% in both tests. The tested alternatives retain performance close to the default.
W3. Visualizing segmentation and confirmation
Appendix D visualizes state signals, segment boundaries, confirmation times, and observations within segments along actual trajectories, clarifying the grouping and memory-writing process.
W4. Matched persistent-memory limits
To test whether the advantage persists under a common persistent-memory limit, we capped both FlexMemWAM and Fixed-4 at six closed segments with eight tokens per segment: 48 persistent memory positions per layer. The anchor and recent/open observations are accounted for separately.
| Method | Battery Try | Swap T |
|---|---|---|
| FlexMemWAM | 78% | 75% |
| Fixed-4 | 57% | 65% |
FlexMemWAM achieves 78% success rate on Battery Try, compared with 57% for Fixed-4. On Swap T, FlexMemWAM achieves 75%, compared with 65% for Fixed-4. Thus, increasing the persistent-memory budget is not necessary to retain an advantage over uniform segmentation on these tasks.
Response to Reviewer erFR
We appreciate the reviewer’s suggestions on runtime, segmentation, and anchor dependence. We added latency measurements, broader ablations, and trajectory visualizations to examine the behavior and cost of the memory mechanism more directly.
Q1. Real-robot input latency and component timings
We added software end-to-end measurements using recorded FR3 observations on RTX 4090. Timing covers observation submission, complete inference, and action return. The table also reports the component breakdown. Those columns are per-step means from separate stage measurements. All values are in ms.
Relative to Frame-wise, FlexMemWAM has an 8.4% lower median and 8.4% higher p95. The stage measurements help explain the median reduction: intermittent writing reduces the per-step memory-writing cost, while the extra work occurs at closure. In the FR3 8D stage test, writing and suffix recomputation take 19.30 and 4.34 ms per invocation, respectively.
Each method retains its native input and generation pipeline. LingBot-VA additionally spends 386.46 ms per step on video generation, so its end-to-end figure reflects that pipeline as well as action prediction.
Q2. Shared segmentation configurations
The rules are shared across tasks within each platform. All RMBench tasks use one segmentation configuration, and all FR3 tasks use another. Neither platform uses per-task tuning. Both configurations use the same three types of cues: post-motion stabilization, gripper transitions, and deceleration.
Q3. Threshold and length sensitivity
We tested alternative segment lengths and thresholds to examine whether performance depends narrowly on the default configuration.
Segment length. With eight memory slots per segment (K=8):
Thresholds. Jointly changing the stabilization and gripper-endpoint thresholds gives:
Put Back Block remains at 100% under both changes. Swap T success changes from 76% to 73% with the wider length range and to 74% with the alternative thresholds. The tested alternatives retain performance close to the default. The observed success therefore does not depend on that single parameter setting.
Q4. Representative segments, random boundaries, and failure
Appendix D adds representative segmentation visualizations, a random-boundary comparison on Swap Blocks matched in segment-length distribution, and a Press Button failure case.