- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MOST compresses each sampled multi-view observation into one token, runs those plus K future queries through a block-causal Memory Expert, and gates each memory layer into the matching action-expert layer. The new part is training-only supervision: every token predicts the active subtask and the state of each relevant object from closed-set vocabularies. RoboCasa-Long, three multi-minute tasks with 200 demonstrations each, is introduced alongside progress metrics adapted from PRM-as-a-Judge 1.5. Against memoryless XR-1 and MOST-Raw (history kept, supervision dropped), MOST gains on RoboCasa365 (65.9 to 72.1 SR), RoboCasa-Long (20.83 to 42.22 MP) and a fixed-base Galbot G1 task.
Critical strengths
- The ablation matches the claim. With everything else fixed, Current Task State comes in at 56.45 SR, below No Memory at 57.91, which rules out the instantaneous-estimate reading.
- Cheap to run: one token per timestep, cached blocks, auxiliary heads dropped at inference, no planner or retrieval.
Critical weaknesses
- MOST-Raw carries no auxiliary signal at all, so this compares structured supervision against none, not structured memory against unstructured memory. Almost any dense per-token target would regularize those tokens. I would want MOST-Raw at a matched supervision budget with different content: future feature prediction, action chunks read off the memory tokens, or the same heads on permuted labels. Tab. 4 varies content against content only.
- Thirty rollouts per task, ten trials on hardware, seed 42, no intervals. The 6.66 to 43.33 real-world headline rests on thirty trials, where three extra successes move it ten points, and Fig. 4 reads a growing gap off four-task groups with deltas of a few points. Three seeds or a bootstrap would settle it.
- Nothing shows what the memory holds: no auxiliary-head accuracy on held-out episodes, no decay with token age or with k, no probing. The gates start at zero, so their converged magnitudes are a free readout and go unreported. The counting task is the sharpest test available and is not run. Nothing on K, B, the sampling interval, or the loss weight either, which changes silently across settings.
- Each view collapses to a CLS vector and the three then to one token, so spatial structure is gone before the Memory Expert sees anything, while the heads are asked for local judgements like in hand or on the counter. Sec. 3.2 defends this on memory growth, which is about cost rather than sufficiency; two or four tokens per timestep would show how much headroom is in the encoder. Spatially dynamic computation may be worth a look, optionally: AdaptiveNN (Wang et al., Nature Machine Intelligence 2025) spends computation on task-conditioned fixations instead of pooling flat, and the instruction is already there to condition where to look. A memory frame currently costs three ViT passes every 0.8s.
- RoboCasa-Long is three tasks evaluated in the environments the demonstrations came from, with no held-out kitchen or object, so it tests execution rather than generalization. None of the memory baselines discussed in Sec. 2 is run, the GR00T N1.5 rows are published numbers the text later treats as matched, and the real-robot setup is fixed-base, so the mobile claim holds in simulation only. My reservations sit mostly in 1 and 2.
Questions for authors
See weakness
4 comments
Response to Reviewer 8jLA Part 2
W3B: Learned memory gates
The model has 36 scalar memory-to-action gates, one for each paired Memory Expert and Action Expert layer.
| Split | Number of gates | Mean | g | | Max | g | |
|---|---|---|---|
| RoboCasa365 Composite-Seen | 36 | 0.004161 | 0.017578 |
| RoboCasa365 Composite-Unseen | 36 | 0.005323 | 0.017822 |
| RoboCasa-Long | 36 | 0.004091 | 0.041260 |
The gates start at zero and learn non-zero values, showing that the model activates the memory-to-action pathway during training. RoboCasa-Long has the largest maximum gate value at 0.041260, which suggests that some layers rely more strongly on memory in longer tasks. Each gate scales a learned cross-attention residual, so its value should not be interpreted as a percentage of the final action prediction.
W3C: Counting-based memory evaluation
The paper already evaluates counting-based memory on the real Galbot G1 robot. We use three task variants with 10 trials each, where the robot must track the number of completed pick-and-place actions and stop at the instructed count. MOST succeeds in 13/30 trials, compared with 11/30 for MOST-Raw and 2/30 for XR-1. This directly tests whether the policy can retain accumulated task progress under real perception and control.
We further add two representative RMBench tasks to provide a standardized simulation comparison with existing memory methods. Battery Try tests memory over repeated state-changing attempts, while Press Button requires the robot to track and execute a specified number of button presses. The published comparison includes EventVLA, MemER [1], Mem-0, HAMLET [2], and MemoryVLA.
| Method | Battery Try | Press Button | Two-task Avg. |
|---|---|---|---|
| † | 30 | 3 | 16.5 |
| Uniform Sampling† | 28 | 3 | 15.5 |
| EventVLA† | 35 | 3 | 19.0 |
| MemER† | 27 | 0 | 13.5 |
| Mem-0† | 28 | 0 | 14.0 |
| HAMLET† | 26 | 0 | 13.0 |
| MemoryVLA† | 33 | 0 | 16.5 |
| MOST | 35 | 5 | 20.0 |
† Published results reported by IMPRINT [3]. MOST is evaluated with 100 rollouts per task using the RMBench success criteria.
Within the rebuttal-period compute budget, we selected these two tasks to cover complementary object-state and count-based progress requirements. We report them explicitly as a two-task subset, not as the complete RMBench result. MOST matches the best published Battery Try result and achieves the highest Press Button result and two-task average. Together with the real-robot counting experiment in the paper, these results verify count-based task-progress memory in both physical execution and a standard memory benchmark.
W3D: , block size, sampling interval, and loss weights
These settings are reported in the paper. Appendix A.1 gives the block size and number of future queries . Appendix A.2 gives the loss weights. Appendix A.3 gives the memory sampling intervals. Appendix B.4 explains how future query is supervised with the label at . We collect them in one table below.
| Setting | Block size | Future queries | Memory sampling interval | ||
|---|---|---|---|---|---|
| RoboCasa365 | 8 | 16 | 16 frames / 0.8 s | 0.25 | 0.25 |
| RoboCasa-Long | 8 | 16 | 16 frames / 0.8 s | 1.00 | 1.00 |
| Real robot | 8 | 16 | 30 frames / 1.0 s | 0.75 | 0.75 |
Here, is the number of history tokens in each block-causal attention block. is the number of future queries. Query is supervised with the label at . The sampling interval determines how often a multi-view observation is compressed into one memory token. It is separate from the control frequency.
W4: One-token compression and spatial information
The one-token compression applies only to historical memory. The current multi-view observation is still processed by the vision-language model (VLM) and supplied to the Action Expert for spatial control. The history token is designed to retain persistent semantic state, such as whether an object has been moved, grasped, or placed. It is not intended to reconstruct the exact spatial layout of an earlier frame.
The probe experiment tests whether task-state information remains decodable after compression. It reaches 69.98% and 68.22% on the two RoboCasa365 splits and 85.62% on RoboCasa-Long. A single cached token therefore retains substantial information about the corresponding task state.
The single-token design also controls the cost of long histories. On RoboCasa-Long, end-to-end inference takes 764.52 ms and 10.21 GB. These measurements include all three memory-view encodings and the memory-to-action layers. From 1k to 8k steps, graphics processing unit (GPU) memory increases by only 0.08 GB and latency does not increase with history length. One token per sampled timestep and block-wise caching keep the memory cost stable over multi-minute execution.
Response to Reviewer 8jLA Part 3
W5A: Scope of RoboCasa-Long and generalization
RoboCasa-Long is designed for natural multi-minute household tasks. Existing benchmarks provide many short or purpose-built memory tasks, but few natural household activities that span several work areas and require minutes of continuous execution. RoboCasa-Long provides these tasks, 200 demonstrations per task, and progress-aware evaluation for the community.
RoboCasa-Long is designed to isolate long-horizon execution under controlled task settings. It is not intended as a held-out-kitchen or held-out-object benchmark. We evaluate scene and object generalization on the full RoboCasa365 suite. Across Atomic, Composite-Seen, and Composite-Unseen, the training and evaluation episodes vary kitchen scenes, object categories, and object placements. Composite-Unseen additionally tests unseen task compositions. The two benchmarks therefore serve different roles: RoboCasa-Long tests multi-minute memory, while RoboCasa365 evaluates generalization across scenes and objects.
W5B: Existing memory baselines
Our original baselines isolate the source of the gain: XR-1 measures the effect of adding memory to the same backbone, while MOST-Raw keeps the same history and architecture but removes task-state supervision. To further compare with an existing memory design, we directly train HAMLET, a history-aware vision-language-action (VLA) policy [2], unchanged on all three RoboCasa-Long tasks. It uses the same demonstrations, training budget, and 30-rollout-per-task protocol. The table reports averages over all three tasks.
| Method | SR | MC@50 | MP | OPS | UPS |
|---|---|---|---|---|---|
| XR-1 | 0.00 | 3.33 | 20.83 | 14.92 | 30.55 |
| MOST-Raw | 0.00 | 20.00 | 28.75 | 22.71 | 46.16 |
| HAMLET | 0.00 | 17.77 | 25.74 | 20.96 | 34.23 |
| MOST | 6.66 | 34.44 | 42.22 | 38.77 | 76.17 |
MOST outperforms HAMLET on every reported metric. It improves MP by 16.48 points and UPS by 41.94 points across the complete RoboCasa-Long benchmark. The RMBench table above additionally provides comparisons with MemoryVLA, MemER, Mem-0, and EventVLA.
W5C: Status of the GR00T N1.5 numbers
GR00T N1.5 follows the same official RoboCasa365 protocol used in our evaluation. The training-data splits, target tasks, and evaluation scenes are the same. GR00T therefore provides a fair benchmark-level memoryless reference and is not used in the controlled memory ablation.
W5D: Fixed-base real-robot evaluation
The real-robot base is intentionally fixed to isolate memory-dependent counting and stopping from navigation failures. The robot rotates its waist between the pickup and placement workspaces. This preserves repeated viewpoint and workspace changes while preventing navigation errors from affecting the memory comparison. The real-robot experiment evaluates memory under physical perception and control. Mobile navigation and manipulation are evaluated in simulation.
References
[1] Sridhar, A., et al. “MemER: Scaling Up Memory for Robot Control via Experience Retrieval.” ICLR, 2026. arXiv:2510.20328.
[2] Koo, M., et al. “HAMLET: Switch Your Vision-Language-Action Model into a History-Aware Policy.” ICLR, 2026. arXiv:2510.00695.
[3] Wu, Y., et al. “Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies.” arXiv:2610.14406, 2026.
I thank the authors for their detailed responses! After reading them in detail, I decided to maintain my positive score.
Response to Reviewer 8jLA Part 1
Response summary
Thank you for recognizing that our controlled ablation matches the central claim of the paper. In particular, the Current Task State variant performing below No Memory rules out the explanation that MOST only estimates the instantaneous state. The review also recognizes that one token per timestep, block-wise caching, and removing the auxiliary heads at inference make the method efficient. Together, these observations support both the task-state memory formulation and its practical design. The remaining questions concern additional controls, evaluation details, memory diagnostics, and benchmark scope. They ask for more complete evidence and reporting around the same core idea. We address them with a matched future-feature control, clarified evaluation seeds, cached-token accuracy and gate statistics, counting results on RMBench, complete configuration details, and a directly trained HAMLET baseline.
W1: Matched auxiliary supervision for MOST-Raw
To test whether any dense auxiliary target would provide the same gain, we add MOST-Raw + Future Feature Prediction. It uses the same Memory Expert, future queries, supervised token positions, training schedule, and auxiliary-loss weight as MOST, but predicts future visual features instead of task-progress and object-state labels.
Metrics follow Appendix B.5: success rate (SR), milestone coverage at 50% progress (MC@50), max progress (MP), path-weighted progress length (PPL), ordered progress score (OPS), and unordered progress score (UPS).
Future feature prediction remains close to MOST-Raw and improves only UPS. In contrast, task-state supervision improves every reported metric by a large margin. Future visual features mix persistent task state with changes in viewpoint, background, and object appearance that may not affect the next action. These features can shift across scenes and viewpoints, making them less robust under out-of-distribution observations. Object states and task progress remove these irrelevant variations and directly describe what the policy needs to remember. The result shows that MOST benefits from the content of its supervision, not simply from adding a dense auxiliary loss.
W2: Evaluation protocol and seeds
Seed 42 in Appendix A.2 is the training seed, not the evaluation seed. RoboCasa365 and RoboCasa-Long use evaluation seeds 0-9. We run three rollouts for each seed, giving 30 rollouts per task. Every method uses the same evaluation seeds. The reported simulation results are averages over these rollouts.
For the real-robot evaluation, we run 10 trials per task with inference seeds 0–9 and reset the physical scene before every trial. The final result averages 30 physical trials across the three task variants. We will revise the paper to clearly separate the training seed from the evaluation protocol.
Figure 4 does not introduce a separate set of trials. It groups the same 16 Composite-Seen tasks by demonstration length and reports the average change from XR-1 in each group. It shows larger observed gaps in the longer groups G3 and G4. We use this result only as a horizon-based analysis of the main evaluation.
W3A: What information is stored in memory
We keep the training-time task-state classifier only for this offline analysis and evaluate it under the original test configuration. Its predictions are not used by the policy. We include both successful and failed episodes. We evaluate only finalized historical tokens that have been committed to the cache. Tokens in the unfinished dynamic block are excluded. Each cached token is counted once and compared with the task-state label at its original sampling timestep.
Probe Accuracy is the overall top-1 classification accuracy over valid task-state labels. Balanced Accuracy is the mean recall across classes, which prevents frequent states from dominating the result. The similar probe and balanced accuracy on both RoboCasa365 splits show that the result is not caused by class imbalance. The 85.62% probe accuracy on RoboCasa-Long shows that one cached token can retain the task state needed in multi-minute tasks.
A finalized cache token is fixed after it is stored, so its representation does not decay or get overwritten as it becomes older. Token age can change how strongly the Action Expert uses that token, but not the information stored inside it.