acceptodds
Under review as a conference paper at ICLR 2027

FlexMemWAM: Flexible Temporal Compression for World Action Models

Abstract

Long-horizon robotic manipulation requires world action models (WAM) to retain historical evidence while controlling memory costs. Task-relevant changes are unevenly distributed over time: many consecutive observations are redundant, whereas brief interactions can introduce important new evidence. Keyframe selection may discard such evidence, while fixed per-frame compression assigns the same memory capacity to every observation. We propose FlexMemWAM, which uses online robot-state cues to organize history into variable-length segments, each sharing a fixed set of learnable slots. Once a boundary is causally confirmed, a shared set of learnable slots jointly reads the segment and prior context within the WAM’s video Transformer layers, producing layer-wise key–value memories for subsequent action prediction. The memory slots are optimized through joint video and action prediction. FlexMemWAM achieves average success rates of 84.1% on nine history-dependent RMBench tasks and 71.7% on three real-world memory-required tasks, outperforming other strong competitors. We also show that FlexMemWAM effectively reduces the historical KV story compared with per-frame allocation. The code will be made publicly available.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-Action Models, Memory Compression, Adaptive Temporal Segmentation, Long-Horizon Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_erFROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

FlexMemWAM manages visual history through state-guided temporal segmentation and learned segment compression. Each variable-length segment shares a fixed set of memory slots, whose layer-wise K/V representations support subsequent action prediction. Experiments on RMBench and three real-world tasks demonstrate promising performance.

Critical strengths
  • Well-motivated memory management. Jointly compressing consecutive observations provides a sensible way to reduce redundant memory allocation while retaining historical evidence.
  • Careful causal design. Boundary confirmation, attention visibility, and cache updates are explicitly specified, supporting online execution without requiring future information.
  • Effective learned compression. Segment memory is integrated into the native WAM layers and optimized through joint video and action prediction.
  • Component ablations support the contributions of the initial anchor and segment memory, while simulation and real-world evaluations cover several forms of history dependence.
  • The paper clearly distinguishes cache-capacity analysis from runtime evaluation and acknowledges the limitations of predefined segmentation signals.
Critical weaknesses
  • Missing real-world inference latency evaluation. Cache savings alone do not establish practical efficiency. Segment writing and suffix recomputation introduce additional computation, but their effects on end-to-end latency and control responsiveness are not measured.
  • Hand-crafted grouping with insufficiently established generality. Although compression is learned, segmentation and write triggering rely on predefined robot-state cues, thresholds, and length constraints. These are reasonable for the evaluated tasks, but may miss important visual changes without corresponding robot motion.
  • Insufficient visualization of variable-length segments. The grouping and writing procedure is relatively complex. Actual trajectories showing state signals, boundaries, confirmation delays, and segment lengths would help establish whether the resulting groups align with meaningful interactions.
  • The ablation study covers only two tasks and could more thoroughly examine adaptive segmentation. Although Fixed-4 provides a uniform-segmentation baseline, the comparison does not control for total cache capacity: using eight slots per segment yields different memory budgets when the number of segments differs. Comparing adaptive and uniform segmentation under matched total cache budgets would better isolate the benefit of boundary placement from differences in memory allocation.
Questions for authors
  • Could you report real-robot end-to-end latency, including median and tail latency, with a breakdown of encoding, memory writing, suffix recomputation, and action denoising? How does this compare with per-frame compression and LingBot-VA?
  • Are the same segmentation rules and hyperparameters used across all simulation and real-world tasks, or are they adjusted for individual tasks?
  • How sensitive is performance to the segmentation thresholds and segment-length constraints?
  • Could you visualize representative segmentations, including failure cases, and compare against random boundaries matched to the proposed method’s segment-length distribution?
  • Removing the initial anchor causes substantial performance drops on the two ablated tasks. Why is the model so dependent on the anchor, and does this dependence hold across the other tasks? Could you provide further analysis and broader ablations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer erFR

We appreciate the reviewer’s suggestions on runtime, segmentation, and anchor dependence. We added latency measurements, broader ablations, and trajectory visualizations to examine the behavior and cost of the memory mechanism more directly.

Q1. Real-robot input latency and component timings

We added software end-to-end measurements using recorded FR3 observations on RTX 4090. Timing covers observation submission, complete inference, and action return. The table also reports the component breakdown. Those columns are per-step means from separate stage measurements. All values are in ms.

MethodSoftware end-to-end latency (ms)Component means (ms/step; separate measurements)
p50p95EncodingMemory writingSuffix KV recomputationAction denoising
FlexMemWAM189.73226.9122.085.991.35130.33
Frame-wise207.06209.4222.0219.08N/A131.12
LingBot-VA914.461117.9253.34N/AN/A391.61

Relative to Frame-wise, FlexMemWAM has an 8.4% lower median and 8.4% higher p95. The stage measurements help explain the median reduction: intermittent writing reduces the per-step memory-writing cost, while the extra work occurs at closure. In the FR3 8D stage test, writing and suffix recomputation take 19.30 and 4.34 ms per invocation, respectively.

Each method retains its native input and generation pipeline. LingBot-VA additionally spends 386.46 ms per step on video generation, so its end-to-end figure reflects that pipeline as well as action prediction.

Q2. Shared segmentation configurations

The rules are shared across tasks within each platform. All RMBench tasks use one segmentation configuration, and all FR3 tasks use another. Neither platform uses per-task tuning. Both configurations use the same three types of cues: post-motion stabilization, gripper transitions, and deceleration.

Q3. Threshold and length sensitivity

We tested alternative segment lengths and thresholds to examine whether performance depends narrowly on the default configuration.

Segment length. With eight memory slots per segment (K=8):

Length rangePut Back BlockSwap T
3–8, default100%76%
2–10100%73%

Thresholds. Jointly changing the stabilization and gripper-endpoint thresholds gives:

ThresholdsPut Back BlockSwap T
Stabilization 0.95; gripper 0.2/0.8, default100%76%
Stabilization 0.9; gripper 0.1/0.9100%74%

Put Back Block remains at 100% under both changes. Swap T success changes from 76% to 73% with the wider length range and to 74% with the alternative thresholds. The tested alternatives retain performance close to the default. The observed success therefore does not depend on that single parameter setting.

Q4. Representative segments, random boundaries, and failure

Appendix D adds representative segmentation visualizations, a random-boundary comparison on Swap Blocks matched in segment-length distribution, and a Press Button failure case.

AuthorsOpenReview ↗

Response to Reviewer erFR

Q5. Anchor dependence across tasks

The anchor provides an initial-scene reference for both memory writing and action prediction. For example, Put Back Block requires remembering the block's location before it is moved. To test whether the dependence extends beyond the original two tasks, we added Press Button and Put Back Block:

TaskFull modelWithout anchor
Swap T76%16%
Cover Blocks100%61%
Press Button67%26%
Put Back Block100%33%

Removing the anchor lowers success across all four tasks. These tasks include both those requiring a single critical historical state and those requiring history across multiple interactions. The dependence therefore extends beyond the two original ablations.

The anchor and segment memory provide complementary information: the initial scene and the subsequent interaction history. On Swap T and Cover Blocks, removing segment memory reduces success even when the anchor is retained. The benefit of segment memory therefore extends beyond simply preserving the initial observation.

W1. Runtime efficiency and responsiveness

We added execution-oriented timing to examine the practical effect of memory writing and suffix recomputation. On RTX 4090 with recorded FR3 inputs, software end-to-end p50/p95 is 189.73/226.91 ms for FlexMemWAM, 207.06/209.42 ms for Frame-wise, and 914.46/1117.92 ms for LingBot-VA.

Relative to Frame-wise, the median is 8.4% lower and p95 is 8.4% higher. In the separate FR3 8D stage test, encoding, memory writing, suffix recomputation, and action denoising average 22.08, 5.99, 1.35, and 130.33 ms per step, respectively. Intermittent writing reduces average per-step memory-writing cost, while writing and recomputation at segment closure add computation, resulting in lower median but higher tail latency.

W2. Generality of state-based grouping

The robot-state cues determine when a segment is closed. They do not restrict its contents to observations that trigger motion events. A sampled visual change remains accessible through the recent/open context and participates in joint compression at closure, even if it does not create a boundary itself. Long-term retention without robot motion remains untested.

To examine the value of observations within a segment, we compared joint compression with an uncompressed segment-end keyframe baseline on Battery Try. The baseline uses the same boundaries and directly inputs segment endpoints plus the initial and current observations. FlexMemWAM achieves 83% success rate, compared with 71% for the keyframe baseline. This shows that retaining endpoints alone does not recover the benefit of jointly represented segment history.

The configuration is shared within each platform: one for RMBench and another for FR3, without per-task tuning. Changing stabilization from 0.95 to 0.9 and gripper thresholds from 0.2/0.8 to 0.1/0.9 changes Swap T from 76% to 74%. Widening segment lengths from 3–8 to 2–10 changes it from 76% to 73%. Put Back Block remains at 100% in both tests. The tested alternatives retain performance close to the default.

W3. Visualizing segmentation and confirmation

Appendix D visualizes state signals, segment boundaries, confirmation times, and observations within segments along actual trajectories, clarifying the grouping and memory-writing process.

W4. Matched persistent-memory limits

To test whether the advantage persists under a common persistent-memory limit, we capped both FlexMemWAM and Fixed-4 at six closed segments with eight tokens per segment: 48 persistent memory positions per layer. The anchor and recent/open observations are accounted for separately.

MethodBattery TrySwap T
FlexMemWAM78%75%
Fixed-457%65%

FlexMemWAM achieves 78% success rate on Battery Try, compared with 57% for Fixed-4. On Swap T, FlexMemWAM achieves 75%, compared with 65% for Fixed-4. Thus, increasing the persistent-memory budget is not necessary to retain an advantage over uniform segmentation on these tasks.

Reviewer_jkidOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlexMemWAM, a memory mechanism for world–action models (WAMs) that compresses visual history into variable-length segments. Online robot-state cues, namely post-motion stabilization, gripper transitions, and deceleration, determine causally confirmed segment boundaries. Each closed segment is then jointly compressed into a fixed set of learnable slots within the native video-Transformer layers. The action expert reads these memories together with the anchor and the recent and still-open observations. Across experiments, FlexMemWAM shows 84.1% average success on nine RMBench tasks and 71.7% on three real-world tasks. In a synthesized cache analysis, it also reduces historical KV storage relative to per-frame allocation.

Critical strengths

Strengths

  • Grouping temporally redundant observations into adaptive segments, rather than compressing each frame independently or selecting sparse keyframes, is an intuitive and well-motivated way to retain historical evidence at lower memory cost.
  • Evaluation spans simulation and real-world tasks. Ablations show that both the initial anchor and the segment memory are important.
Critical weaknesses

Weaknesses

[W1] Limited backbones and benchmarks. All experiments use a single Wan2.2-based WAM backbone, and a separate model is trained for each RMBench task. It is therefore unclear whether adaptive segment memory transfers to other WAMs. Evaluation is also limited to nine RMBench tasks and three short real-world tasks, with no results on other memory benchmarks, longer-horizon settings, or standard non-memory benchmarks.

[W2] Unclear advantage over existing memory baselines (e.g., MemoryWAM). MemoryWAM, the closest per-frame compression baseline, is limited; it is absent from the real-world comparison. The average improvement is marginal (84.1% vs. 83.0%) and comes almost entirely from a single task, Battery Tr, and its M(1) average is lower (81.4% vs. 84.2%). Moreover, the main claimed benefit over per-frame compression is efficiency, yet only synthesized KV capacity is reported. It therefore remains unclear what practical advantage FlexMemWAM offers over existing memory baselines.

Questions for authors

Questions

[Q1] Can adaptive segment memory be applied to another WAM backbones? Does it remain effective on other memory benchmarks (e.g., Robodojo), or on longer-horizon tasks?

[Q2] Under identical training conditions, how does FlexMemWAM compare with MemoryWAM in task success, inference latency, and memory usage? Why does it underperform on Press Button and Swap T?

[Q3] How sensitive is performance to the segmentation design, such as the segment length range (L_min, L_max), the number of slots K, and each of the three boundary signals?

[Q4] What is the end-to-end inference latency, including memory writing and suffix recomputation at each boundary? What is the training cost relative to per-frame compression and to the backbone without memory?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jkid

We appreciate the reviewer’s comments on evaluation breadth and the practical benefits over memory baselines. We expanded the backbone and benchmark coverage and added sensitivity studies and inference-cost measurements to clarify where segment memory helps.

Q1. Additional backbone, benchmark, and task horizons

We have expanded the evaluation to a second backbone and an additional memory benchmark. The existing tasks also include sustained, history-dependent operation.

Another backbone. We applied segment memory to Cosmos Predict 2.5-2B:

Backbone with segment memoryPut Back BlockSwap T
Wan2.2100%76%
Cosmos Predict 2.5-2B100%65%

The method reaches 100% on Put Back Block and 65% on Swap T with Cosmos, demonstrating applicability beyond the Wan2.2 backbone.

Another memory benchmark. On RoboDojo Memory, we implemented FlexMemWAM on top of OpenWAM-α and compared it with the officially reported OpenWAM-α baseline:

MethodSuccess rate
OpenWAM-α (officially reported)9.11%
FlexMemWAM (based on OpenWAM-α)24.67%

FlexMemWAM improves success from 9.11% to 24.67%, extending the evaluation of segment memory beyond RMBench.

Task duration. RMBench Blocks Ranking Try lasts approximately 50 seconds, with 100% success rate. Real-robot Press Button lasts approximately one minute and requires tracking the press count across repeated interactions. These evaluations already test the use of history over sustained multi-step operation.

Q2. MemoryWAM comparison and task-specific gaps

The comparison shows similar overall success and median latency, with lower historical storage for FlexMemWAM.

Success and inference cost. We reproduced MemoryWAM and report its average RMBench success below. We measured complete planning latency and historical KV storage on RTX 4090 using our implementations. H denotes the number of planning observations. H=112 is a synthetic long-history stress test.

MetricFlexMemWAMMemoryWAM
Average RMBench success rate84.1%83.0%
Planning p50, H=2–40203.42 ms207.83 ms
Planning p95, H=2–40243.83 ms210.89 ms
Historical KV across all layers, H=112438.75 MiB562.50 MiB

FlexMemWAM uses less historical KV storage with comparable overall success and median planning latency. The additional computation is concentrated at segment closure.

Task-specific gaps. On Press Button, FlexMemWAM achieves 67%, compared with the 87% reported in the MemoryWAM paper. The task requires counting effective presses. A shared segment representation may make visually similar repeated events harder to distinguish. On Swap T, FlexMemWAM achieves 76%, compared with MemoryWAM's paper-reported 94% using additional action-history conditioning (Appendix A.1 of MemoryWAM). FlexMemWAM does not use historical actions. We additionally reimplemented MemoryWAM without action-history conditioning. This baseline achieves 58% success rate. FlexMemWAM achieves a higher success rate of 76% without using action history.

Q3. Sensitivity to segment length, slots, and triggers

We examined segment length, memory capacity, and dependence on the three closure cues separately.

Segment length. With K=8:

Segment-length rangePut Back BlockSwap T
3–8100%76%
2–10100%73%

Widening the length range leaves Put Back Block at 100% and changes Swap T success from 76% to 73%. Performance remains close to the default.

Memory capacity. With segment lengths fixed to 3–8:

Slots per segment KPut Back BlockSwap T
4100%69%
8 (default)100%76%
16100%77%

Swap T success increases from 69% at K=4 to 76% at K=8 and 77% at K=16. Put Back Block remains at 100%. The default eight slots therefore achieve performance close to sixteen slots while using half as many slots per segment.

Closure cues. We disabled one boundary-trigger signal at a time, keeping the remaining segmentation rules and memory writing and reading active. Disabling stabilization, gripper transitions, and deceleration yields Swap T success rates of 72%, 65%, and 71%, respectively, versus 76% using all signals. Put Back Block remains at 100% in all three conditions. Swap T is most sensitive to removing the gripper-transition trigger, whereas Put Back Block shows no success-rate change under the tested ablations.

AuthorsOpenReview ↗

Response to Reviewer jkid

Q4. Complete inference latency and training cost

Complete inference. On RTX 4090 with batch size 1, BF16, and 10 action-denoising steps, complete planning over histories of 2–40 observations takes 211.08 ms on average, with 203.42 ms median and 243.83 ms p95. This includes encoding, memory writing, suffix KV recomputation when needed, and action denoising. In a separate trajectory replay, closing and ordinary steps have medians of 240.76 and 201.96 ms.

Training computation. Segment sharing reduces memory slots and associated attention computation relative to frame-wise compression, while historical source observations still require encoding and backpropagation. Relative to memory-free Fast-WAM, the additional computation comes from historical encoding and memory reads/writes with gradient propagation. Training cost therefore depends on history length and segmentation.

W1. Breadth of evaluation

We have broadened the evaluation along the three dimensions raised by the reviewer.

Backbone and memory benchmark. With Cosmos Predict 2.5-2B, segment memory achieves 100% and 65% on Put Back Block and Swap T. On RoboDojo Memory, FlexMemWAM built on OpenWAM-α achieves 24.67% success rate, outperforming the officially reported OpenWAM-α baseline of 9.11%. These experiments extend the evaluation beyond both Wan2.2 and RMBench.

Standard manipulation. On standard LIBERO, FlexMemWAM achieves 96.2%, close to Fast-WAM's reported 97.6%, retaining standard manipulation performance while incorporating segment memory.

Sustained operation. Existing evaluations include approximately 50-second Blocks Ranking Try and one-minute real-robot Press Button, the latter requiring press-count tracking across repeated interactions. Thus, the evaluation includes sustained use of history as well as additional backbone and benchmark settings.

W2. Practical benefits relative to memory baselines

1. Performance advantages. FlexMemWAM and our MemoryWAM reproduction achieve nine-task averages of 84.1% and 83.0%, respectively. The task-level results clarify the gains: MemoryWAM already reaches 100% on Rearrange Blocks, Put Back Block, Swap Blocks, and Blocks Ranking, and 98% on Cover Blocks, leaving little room for improvement on five tasks. FlexMemWAM maintains all four perfect scores and raises Cover Blocks to 100%. It also improves Observe and Pick Up (31% versus 27%) and Battery Try (83% versus 41%) relative to MemoryWAM's paper-reported results.

On Swap T, MemoryWAM reports a 94% success rate with additional action-history conditioning (Appendix A.1 of MemoryWAM). FlexMemWAM does not use action history. We reimplemented MemoryWAM without this conditioning. This baseline achieves 58% success rate. FlexMemWAM achieves a higher success rate of 76%. With this comparison on Swap T, FlexMemWAM improves success on four tasks while maintaining 100% success rate on four additional tasks.

We also evaluated MemoryWAM on real-robot FR3 Sequential Uncovering:

MethodSequential Uncovering success rate
MemoryWAM (our evaluation)60%
FlexMemWAM75%

FlexMemWAM achieves 75% success rate, compared with 60% for MemoryWAM. This task requires remembering occluded object locations and execution order. Across the full three-task FR3 evaluation, FlexMemWAM averages 71.7%, compared with 60.0% for LingBot-VA and 5.0% for Fast-WAM.

2. Storage efficiency. On RTX 4090, historical KV at H=112 is 438.75 MiB for FlexMemWAM and 562.50 MiB for our MemoryWAM implementation, a 22.0% reduction. Here H denotes the number of planning observations, and H=112 is a synthetic stress test. Segment memory reduces historical storage while retaining directly accessible open-segment observations. Over H=2–40, complete planning p50/p95 is 203.42/243.83 ms versus 207.83/210.89 ms, respectively. These measurements demonstrate lower historical storage with comparable median planning latency.

Thank you for the response. The additional backbone and benchmark evaluations, and sensitivity results address most of W1 and Q3. However, my concern regarding the performance advantage over MemoryWAM remains. In particular, the Press Button gap (67% vs. 87%) involves neither extra component such as action conditioning, so it is a clean comparison. It suggests that the shared segment representation structurally limits counting, a core memory capability. Based on this, I maintain my current score.

Reviewer_VZuJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FlexMemWAM, which uses robot-state signals to define variable-length historical segments and jointly compresses each segment into layer-wise KV memories through a fixed number of learnable slots within the video Transformer. Action prediction uses the initial-frame anchor, compressed segment memories, and uncompressed recent observations and the open segment. The main contribution is to extend the unit of historical compression from individual frames to state-guided, variable-length segments.

Critical strengths
  1. The paper changes the unit of memory allocation from individual frames to robot-state-guided, variable-length segments, allowing consecutive observations to share a fixed capacity and addressing redundancy in frame-wise compression. The boundary-confirmation and open-segment retention mechanisms also account for the causal constraints of online execution.

  2. The experiments provide support for the contribution of compressed memory. In the ablation study, removing segment memory while retaining the same initial-frame anchor and recent/open observations substantially reduces success rates. This indicates that the performance gains cannot be attributed entirely to uncompressed context, strengthening the credibility of the method.

Critical weaknesses
  1. The relevance of the rule-based adaptive segmentation to task requirements is insufficiently validated. Motion stabilization, gripper transitions, and deceleration are used to determine segment boundaries, but these signals do not necessarily correspond to task-relevant changes.

  2. The method imposes no upper bound on the number of accumulated segments and provides no mechanism for updating or consolidating existing memories. Consequently, the total KV-cache size and historical attention cost continue to grow with task length.

Questions for authors
  1. Different segmentation strategies change the length of the open segment and thus the amount of uncompressed history directly accessible to the model. Does the comparison in Table 2 control for this difference? If not, how can the gains from adaptive boundary selection be distinguished from those due to retaining more uncompressed context?

  2. Have you conducted experiments to identify which segment memories are most critical to task success? For example, do memories from segments closed by motion stabilization, gripper transitions, or deceleration differ in their contributions to model performance?

  3. The paper states that segment memories are continually appended without a fixed total capacity. When memory writing, suffix KV recomputation, and action denoising are included, how do end-to-end latency and peak GPU memory usage scale with history length compared with frame-wise compression? Do actual task executions show that the storage savings demonstrated by the synthesized cache analysis translate into practical runtime efficiency gains?

2 comments
AuthorsOpenReview ↗

Response to Reviewer VZuJ

We appreciate the reviewer’s questions about the role of segmentation and its practical efficiency. We added controls for directly accessible context and persistent memory, examined the contribution of individual boundary-trigger signals, and measured full-model storage and latency.

Q1. Controlling access to uncompressed history

Table 2 controls the initial anchor and four-observation recent window, but the open-segment length varies with the segmentation strategy. To test whether additional directly accessible context accounts for the gains, we kept the FlexMemWAM checkpoint and memory-writing procedure fixed and restricted the action branch to the most recent four non-anchor observations.

SettingSwap TCover Blocks
FlexMemWAM, original76%100%
FlexMemWAM, direct access limited to four recent non-anchor observations73%98%
Fixed-4, original Table 2 setting65%92%

The restriction reduces Swap T success from 76% to 73% and Cover Blocks success from 100% to 98%. Both results remain above the original Fixed-4 reference. This indicates that access to additional open-segment observations is not the sole source of the gains.

Q2. Contribution of boundary-trigger signals

According to your helpful suggestion, we added experiments to study the effect of different boundary trigger signals. For this ablation, we disabled each signal as a boundary trigger in turn, keeping the remaining triggers and other segmentation rules active. Segment-memory writing and reading remained unchanged.

Segmentation settingSwap T
Our default setting76%
Without stabilization trigger72%
Without gripper-transition trigger65%
Without deceleration trigger71%

On Swap T, the full model achieves 76% success rate. Disabling stabilization, gripper transitions, or deceleration as boundary triggers reduces success to 72%, 65%, and 71%, respectively. Gripper transitions have the largest observed effect, supporting their role in effective segmentation on this task.

W1. Relationship between boundaries and task-relevant changes

We examined the relevance of state-guided segmentation through boundary-trigger ablations and trajectory visualizations.

Disabling stabilization, gripper transitions, or deceleration as boundary triggers yields Swap T success rates of 72%, 65%, and 71%, respectively, compared with 76% using all signals. Memory writing and reading remain active in every setting. Removing the gripper-transition trigger has the largest observed effect, indicating that it contributes to effective grouping on Swap T. Put Back Block remains at 100% in all three conditions, showing task-dependent sensitivity to the segmentation signals.

The trajectory visualizations in Appendix D further illustrate the relationship between segment boundaries and actual interactions.

AuthorsOpenReview ↗

Response to Reviewer VZuJ

Q3. Storage and runtime as history grows

We added full-model measurements to test both parts of this question: how storage and latency change with history length, and whether the benefit appears during execution.

History length. We tested FlexMemWAM and Frame-wise on RTX 4090 with the same input settings. Timing includes encoding, memory writing, suffix KV recomputation, and action denoising. H denotes the number of planning observations. This test covers synthetic histories of H=1–112. The table reports three representative lengths.

HistoryFlexMemWAMFrame-wise
HTotal historical KV (MiB)Median latency (ms)Total historical KV (MiB)Median latency (ms)
16216.56202.50253.13207.83
40230.63201.29320.63207.06
112438.75203.21523.13211.32

FlexMemWAM dynamically adjusts how history is represented: closed segments form compact memories, while observations from the ongoing segment remain uncompressed and directly accessible to action prediction. At H=112, compressed-memory KV decreases from 312.19 MiB with frame-wise compression to 59.06 MiB. Meanwhile, FlexMemWAM retains four additional uncompressed observations in the open segment, resulting in total historical KV of 438.75 versus 523.13 MiB, a 16.1% reduction. Total KV savings therefore vary with both the accumulated compressed history and the current open-segment length.

Whole-model GPU memory also includes weights and intermediate tensors, while inference time includes encoding, action denoising, and segment-closure maintenance. Their reductions are consequently smaller than the reduction in compressed-memory KV. Across H=1–112, measured peak allocated GPU memory is 14.01 GiB for FlexMemWAM and 14.67 GiB for Frame-wise. At H=112, median planning latency is 203.21 versus 211.32 ms. This reduces historical storage while maintaining comparable planning latency.

Execution latency. With recorded FR3 inputs, software end-to-end p50/p95 is 189.73/226.91 ms for FlexMemWAM and 207.06/209.42 ms for Frame-wise: an 8.4% lower median and 8.4% higher p95. Intermittent writing reduces average per-step memory-maintenance cost, with additional computation concentrated at segment closure, resulting in lower median but higher tail latency. In an additional closed-loop simulation timing test, mean model-service latency is comparable at 194 ms versus 198 ms.

W2. Bounding persistent memory

We evaluated a variant with persistent segment memory capped at 32, 48 and 64 slots per layer. Each closed segment contributes eight slots. When the budget is exceeded, the oldest segment memory is evicted. The initial anchor and recent/open context remain unchanged and are stored separately.

SettingBattery TrySwap T
Our default setting83%76%
32 slots72%68%
48 slots78%75%
64 slots82%76%

Success improves as the persistent-memory budget increases from 32 to 64 slots per layer. With 48 slots, success is 78% on Battery Try and 75% on Swap T, compared with 83% and 76% for the full-history model. With 64 slots, Battery Try reaches 82%, close to the full-history result of 83%. Swap T matches the full-history model at 76%. These results show that a simple eviction policy can bound persistent segment memory while retaining performance close to the full-history model on these tasks.

Reviewer_4hDROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper looks at how to make the frame history cheaper for memory and keeps useful information while grouping and compressing.

The main argument is that evidence is unevenly spread: in some sequences several frames can be redundant, while others can have a high density of information.

The main contribution is FlexMemWAM, a world action model. The model segments the history using robot-state signals (post-motion stabilization, gripper transition, and deceleration), compresses the frames in the segment into a fixed number of tokens, and predicts a sequence of actions with the memory tokens, the first frame, and the last frames (read through attention at every layer). The cache grows linearly like the per-frame method, but uses less memory with the same number of frames.

Critical strengths

The critical strengths of this paper are the grouping and segmentation methods. The SOTA models either compress all the frames from the history or pkeep selected frame. This method aims to get rid of redundant frames and also understand highly dense sequences of information. The cache fills up more slowly. Also, what is interesting is that FlexMemWAM is based on Fast-WAM and performs significantly better (going from 5.9% to 84.1% on RMBench, memory matter), and the ablations on Swap T and Cover Blocks clearly show the importance of the anchor and the segment memory.

Critical weaknesses

For the experiments in the RMBench tasks, we see that FlexMemWAM performs well in Battery Try, but it is close to MemoryWAM (which uses the per-frame method) in the other tasks. On the M(n)M^{(n)} tasks (multiple historical memories), its lead mainly comes from Battery Try, while on average across all tasks it is very close to the per-frame framework (84.1% vs 83.0%). In the ablation part (on Swap T and Cover Blocks), they already compare with the per-frame method (Frame-wise), so it would be interesting to also test that per-frame ablation on Battery Try, and to compare with a keyframe method since it is compared at the beginning of the paper.

The critical weaknesses of this paper are the fixed segmentation rule and the fixed memory capacity. Indeed, the segmentation rule is hand-crafted and not learned, and it only looks at the robot's own movement. If an important visual event happens in the scene while the robot is not moving, the model will not create a segment boundary and so it may be compressed with other visual cues . Also, every segment has a flenght of 3 to 8 frames and is always compressed into 8 tokens, which can lose details in complex phases, and the memory cache still grows linearly with time instead of having a fixed total budget. However, these limitations are listed in the "conclusions and limitations" section . Finally, the cache experiment only measures memory in MiB and does not measure the real computation time when writing memory and recomputing frames.

Questions for authors

Clarification : What do Frame-wise and Fixed-4 score on Battery Try and Press Button?

Future work : Why did you not compare your method with a keyframe baseline in the ablation study, since you compare against keyframes at the beginning of the paper?

What is the real inference time (latency) when a segment closes and you have to write the memory and recompute the open frames?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4hDR

We appreciate the reviewer’s suggestions on the source of the gains and the cost of segment memory. We added task-specific baselines, capacity and memory-budget ablations, and direct measurements of memory writing and suffix recomputation.

Q1. Additional ablations on Battery Try and Press Button

Thanks for your insightful suggestion. We added experiments to compare the three different settings on Battery Try and Press Button. The results are as follows:

MethodBattery TryPress Button
Frame-wise71%64%
Fixed-458%51%
FlexMemWAM83%67%

According to the table, FlexMemWAM consistently outperforms these two alternatives on both tasks.

Q2. Keyframe baseline

Thank you for this valuable suggestion. At the time of submission, we were not aware of a peer-reviewed keyframe-memory method for world action models, and our original ablations focused on frame-wise and segment-wise compression. We agree that an explicit keyframe comparison would strengthen the empirical support for our motivation.

Following your suggestion, we implemented a segment-end keyframe baseline using the same WAM backbone. It uses the same segment boundaries as FlexMemWAM but retains only the final observation of each closed segment, together with the initial and current observations. The selected observations are retained without compression into memory slots.

MethodHistorical representationBattery TryPress Button
Segment-end keyframesUncompressed segment-end observations71%42%
FlexMemWAMJointly compressed segment observations83%67%

FlexMemWAM achieves 83% success rate on Battery Try, compared with 71% for the segment-end keyframe baseline. On Press Button, FlexMemWAM achieves 67%, compared with 42% for the baseline. Even when the selected endpoints are retained without segment compression, this baseline falls short of FlexMemWAM on both tasks. These results support the benefit of our segment-memory design over endpoint-only historical retention on these tasks.

Q3. Latency when a segment closes

We measured end-to-end planning latency on an NVIDIA RTX 4090, including observation encoding, cache updates, and action denoising. The results are:

Planning stepMedian latencyp95 latency
Ordinary step201.96 ms216.06 ms
Segment-closure step240.76 ms254.93 ms

The median latency at segment closure is 38.80 ms higher than at ordinary steps.

In a separate component-level benchmark using the RMBench dual-arm setup, we measured memory writing and suffix KV recomputation with mixed segment lengths and a confirmation delay of one latent frame. Their mean costs are 19.97 ms and 22.36 ms per invocation, respectively. Since these operations are triggered only at segment closure, their costs amortized over all planning steps are 3.78 ms and 4.23 ms, respectively, totaling 8.01 ms per planning step.

AuthorsOpenReview ↗

Response to Reviewer 4hDR

W1. Advantages over per-frame compression

MemoryWAM already achieves 100% on four tasks (Rearrange Blocks, Put Back Block, Swap Blocks, Blocks Ranking) and 98% on Cover Blocks, leaving little room for improvement on five of the nine tasks. FlexMemWAM matches all four perfect scores and raises Cover Blocks to 100%.

Our FlexMemWAM also improves Observe and Pick Up (31% vs. 27%) and Battery Try (83% vs. 41%). On Battery Try, additional comparisons within our WAM framework yield 71% for frame-wise compression, 58% for fixed-length segmentation, and 71% for segment-end keyframes, compared with 83% for FlexMemWAM.

For Swap T, MemoryWAM adds action-history conditioning that is not used on the other tasks (MemoryWAM, Appendix A.1), achieving the reported 94%. FlexMemWAM uses the same conditioning design across all tasks, without introducing task-specific action-history inputs. We additionally reimplemented MemoryWAM without action-history conditioning. This baseline achieves 58% success rate on Swap T. FlexMemWAM achieves a higher success rate of 76% without using action history.

With the comparison on Swap T made without action-history conditioning, FlexMemWAM improves success on four tasks while retaining all four 100% scores.

Besides, the historical KV storage is also lower. At 56 planning observations, our synthesized cache analysis gives 241.9 MiB for segment-level allocation versus 354.4 MiB for frame-wise allocation (31.7% reduction). Both totals include compressed memory and retained uncompressed observations.

W2. Segmentation rules, capacity, and computation

We address the grouping rule, per-segment capacity, and total memory budget separately.

Visual information within a segment. Robot-state cues determine segment boundaries and write timing. All sampled observations assigned to a segment participate in its joint compression, and recent/open observations remain directly accessible before closure. A visual event can therefore enter memory without triggering a boundary itself. Long-term retention without robot motion remains untested.

Per-segment capacity. With segment lengths fixed to 3–8, Swap T success increases from 69% at K=4 to 76% at K=8 and 77% at K=16. Put Back Block remains at 100%. This supports the default eight slots, which approach the performance of sixteen with half as many slots per segment.

Total memory budget. We also evaluated retaining only the six most recent closed segments. This variant achieves 78% success rate on Battery Try and 75% on Swap T. The original model achieves 83% and 76%, respectively. The method thus retains performance close to the original while bounding persistent segment memory.

Measured computation. On RTX 4090, complete planning p50/p95 is 240.76/254.93 ms at closure and 201.96/216.06 ms at ordinary steps, including encoding, cache updates, and action denoising. A separate 14D stage test with mixed segment lengths and a one-observation confirmation delay measures 19.97/22.36 ms per invocation for writing and suffix recomputation. The additional work is concentrated at closure.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and Invitation for Further Discussion

We would like to sincerely thank all reviewers for their time, effort, and constructive feedback throughout the review process. We truly appreciate your valuable comments and suggestions, which have helped us improve the quality and clarity of our paper.

As the author–reviewer discussion period is scheduled to close tonight, we would also like to kindly ask whether you have any remaining questions or concerns. We would be more than happy to provide further clarification or address any additional feedback.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.