acceptodds
Under review as a conference paper at ICLR 2027

MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation

11 views

Abstract

Long-horizon embodied manipulation requires policies to act on information that is no longer visible. Modified objects may leave the field of view, while completed subtasks continue to constrain future actions. Existing approaches extend temporal context through recurrent state updates, test-time training, or storage and retrieval of past observations. As horizons grow, these mechanisms can still leave the policy to infer task state from long, redundant, and viewpoint-dependent visual experience. We argue that long-horizon agents benefit not merely from remembering more, but from remembering task-relevant state changes and progress. We propose Memory of Object States and Task Progress (MOST), a policy-internal memory framework that compresses multi-view observations into temporal tokens. Historical and predictive future tokens are explicitly supervised with object states and subtask progress, structuring memory around what has changed in the world and what has already been accomplished. The resulting memory directly conditions action generation within a single end-to-end policy, without an external planner or retrieval stage. We further introduce RoboCasa-Long, a benchmark of multi-minute, multi-stage mobile manipulation tasks. Across RoboCasa365, RoboCasa-Long, and real-world robot experiments, MOST consistently outperforms Xiaomi-Robotics-1 (XR-1), the previous state-of-the-art model. It raises average success from 65.9% to 72.1% on RoboCasa365, average maximum task progress from 20.83% to 42.22% on RoboCasa-Long, and real-world success from 6.66% to 43.33%. MOST also outperforms a matched visual-history memory variant, showing that effective long-horizon memory depends not only on how much experience is retained, but on whether it preserves decision-relevant progress and world-state changes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Long-Horizon Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8jLAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOST compresses each sampled multi-view observation into one token, runs those plus K future queries through a block-causal Memory Expert, and gates each memory layer into the matching action-expert layer. The new part is training-only supervision: every token predicts the active subtask and the state of each relevant object from closed-set vocabularies. RoboCasa-Long, three multi-minute tasks with 200 demonstrations each, is introduced alongside progress metrics adapted from PRM-as-a-Judge 1.5. Against memoryless XR-1 and MOST-Raw (history kept, supervision dropped), MOST gains on RoboCasa365 (65.9 to 72.1 SR), RoboCasa-Long (20.83 to 42.22 MP) and a fixed-base Galbot G1 task.

Critical strengths
  1. The ablation matches the claim. With everything else fixed, Current Task State comes in at 56.45 SR, below No Memory at 57.91, which rules out the instantaneous-estimate reading.
  2. Cheap to run: one token per timestep, cached blocks, auxiliary heads dropped at inference, no planner or retrieval.
Critical weaknesses
  1. MOST-Raw carries no auxiliary signal at all, so this compares structured supervision against none, not structured memory against unstructured memory. Almost any dense per-token target would regularize those tokens. I would want MOST-Raw at a matched supervision budget with different content: future feature prediction, action chunks read off the memory tokens, or the same heads on permuted labels. Tab. 4 varies content against content only.
  2. Thirty rollouts per task, ten trials on hardware, seed 42, no intervals. The 6.66 to 43.33 real-world headline rests on thirty trials, where three extra successes move it ten points, and Fig. 4 reads a growing gap off four-task groups with deltas of a few points. Three seeds or a bootstrap would settle it.
  3. Nothing shows what the memory holds: no auxiliary-head accuracy on held-out episodes, no decay with token age or with k, no probing. The gates start at zero, so their converged magnitudes are a free readout and go unreported. The counting task is the sharpest test available and is not run. Nothing on K, B, the sampling interval, or the loss weight either, which changes silently across settings.
  4. Each view collapses to a CLS vector and the three then to one token, so spatial structure is gone before the Memory Expert sees anything, while the heads are asked for local judgements like in hand or on the counter. Sec. 3.2 defends this on memory growth, which is about cost rather than sufficiency; two or four tokens per timestep would show how much headroom is in the encoder. Spatially dynamic computation may be worth a look, optionally: AdaptiveNN (Wang et al., Nature Machine Intelligence 2025) spends computation on task-conditioned fixations instead of pooling flat, and the instruction is already there to condition where to look. A memory frame currently costs three ViT passes every 0.8s.
  5. RoboCasa-Long is three tasks evaluated in the environments the demonstrations came from, with no held-out kitchen or object, so it tests execution rather than generalization. None of the memory baselines discussed in Sec. 2 is run, the GR00T N1.5 rows are published numbers the text later treats as matched, and the real-robot setup is fixed-base, so the mobile claim holds in simulation only. My reservations sit mostly in 1 and 2.
Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8jLA Part 1

Response summary

Thank you for recognizing that our controlled ablation matches the central claim of the paper. In particular, the Current Task State variant performing below No Memory rules out the explanation that MOST only estimates the instantaneous state. The review also recognizes that one token per timestep, block-wise caching, and removing the auxiliary heads at inference make the method efficient. Together, these observations support both the task-state memory formulation and its practical design. The remaining questions concern additional controls, evaluation details, memory diagnostics, and benchmark scope. They ask for more complete evidence and reporting around the same core idea. We address them with a matched future-feature control, clarified evaluation seeds, cached-token accuracy and gate statistics, counting results on RMBench, complete configuration details, and a directly trained HAMLET baseline.

W1: Matched auxiliary supervision for MOST-Raw

To test whether any dense auxiliary target would provide the same gain, we add MOST-Raw + Future Feature Prediction. It uses the same Memory Expert, future queries, supervised token positions, training schedule, and auxiliary-loss weight as MOST, but predicts future visual features instead of task-progress and object-state labels.

Metrics follow Appendix B.5: success rate (SR), milestone coverage at 50% progress (MC@50), max progress (MP), path-weighted progress length (PPL), ordered progress score (OPS), and unordered progress score (UPS).

MethodSRMC@50MPPPLOPSUPS
MOST-Raw0.0020.0028.7517.8222.7146.16
MOST-Raw + Future Feature Prediction0.0016.6626.2917.5722.4648.73
MOST6.6634.4442.2233.0338.7776.17

Future feature prediction remains close to MOST-Raw and improves only UPS. In contrast, task-state supervision improves every reported metric by a large margin. Future visual features mix persistent task state with changes in viewpoint, background, and object appearance that may not affect the next action. These features can shift across scenes and viewpoints, making them less robust under out-of-distribution observations. Object states and task progress remove these irrelevant variations and directly describe what the policy needs to remember. The result shows that MOST benefits from the content of its supervision, not simply from adding a dense auxiliary loss.

W2: Evaluation protocol and seeds

Seed 42 in Appendix A.2 is the training seed, not the evaluation seed. RoboCasa365 and RoboCasa-Long use evaluation seeds 0-9. We run three rollouts for each seed, giving 30 rollouts per task. Every method uses the same evaluation seeds. The reported simulation results are averages over these rollouts.

For the real-robot evaluation, we run 10 trials per task with inference seeds 0–9 and reset the physical scene before every trial. The final result averages 30 physical trials across the three task variants. We will revise the paper to clearly separate the training seed from the evaluation protocol.

Figure 4 does not introduce a separate set of trials. It groups the same 16 Composite-Seen tasks by demonstration length and reports the average change from XR-1 in each group. It shows larger observed gaps in the longer groups G3 and G4. We use this result only as a horizon-based analysis of the main evaluation.

W3A: What information is stored in memory

We keep the training-time task-state classifier only for this offline analysis and evaluate it under the original test configuration. Its predictions are not used by the policy. We include both successful and failed episodes. We evaluate only finalized historical tokens that have been committed to the cache. Tokens in the unfinished dynamic block are excluded. Each cached token is counted once and compared with the task-state label at its original sampling timestep.

SplitProbe AccuracyBalanced Accuracy
RoboCasa365 Composite-Seen69.98%70.96%
RoboCasa365 Composite-Unseen68.22%70.71%
RoboCasa-Long85.62%79.10%

Probe Accuracy is the overall top-1 classification accuracy over valid task-state labels. Balanced Accuracy is the mean recall across classes, which prevents frequent states from dominating the result. The similar probe and balanced accuracy on both RoboCasa365 splits show that the result is not caused by class imbalance. The 85.62% probe accuracy on RoboCasa-Long shows that one cached token can retain the task state needed in multi-minute tasks.

A finalized cache token is fixed after it is stored, so its representation does not decay or get overwritten as it becomes older. Token age can change how strongly the Action Expert uses that token, but not the information stored inside it.

AuthorsOpenReview ↗

Response to Reviewer 8jLA Part 2

W3B: Learned memory gates

The model has 36 scalar memory-to-action gates, one for each paired Memory Expert and Action Expert layer.

SplitNumber of gatesMean | g |Max | g |
RoboCasa365 Composite-Seen360.0041610.017578
RoboCasa365 Composite-Unseen360.0053230.017822
RoboCasa-Long360.0040910.041260

The gates start at zero and learn non-zero values, showing that the model activates the memory-to-action pathway during training. RoboCasa-Long has the largest maximum gate value at 0.041260, which suggests that some layers rely more strongly on memory in longer tasks. Each gate scales a learned cross-attention residual, so its value should not be interpreted as a percentage of the final action prediction.

W3C: Counting-based memory evaluation

The paper already evaluates counting-based memory on the real Galbot G1 robot. We use three task variants with 10 trials each, where the robot must track the number of completed pick-and-place actions and stop at the instructed count. MOST succeeds in 13/30 trials, compared with 11/30 for MOST-Raw and 2/30 for XR-1. This directly tests whether the policy can retain accumulated task progress under real perception and control.

We further add two representative RMBench M(n)M(n) tasks to provide a standardized simulation comparison with existing memory methods. Battery Try tests memory over repeated state-changing attempts, while Press Button requires the robot to track and execute a specified number of button presses. The published comparison includes EventVLA, MemER [1], Mem-0, HAMLET [2], and MemoryVLA.

MethodBattery TryPress ButtonTwo-task Avg.
π0.5\pi_{0.5}†30316.5
Uniform Sampling†28315.5
EventVLA†35319.0
MemER†27013.5
Mem-0†28014.0
HAMLET†26013.0
MemoryVLA†33016.5
MOST35520.0

† Published results reported by IMPRINT [3]. MOST is evaluated with 100 rollouts per task using the RMBench success criteria.

Within the rebuttal-period compute budget, we selected these two tasks to cover complementary object-state and count-based progress requirements. We report them explicitly as a two-task subset, not as the complete RMBench M(n)M(n) result. MOST matches the best published Battery Try result and achieves the highest Press Button result and two-task average. Together with the real-robot counting experiment in the paper, these results verify count-based task-progress memory in both physical execution and a standard memory benchmark.

W3D: KK, block size, sampling interval, and loss weights

These settings are reported in the paper. Appendix A.1 gives the block size BB and number of future queries KK. Appendix A.2 gives the loss weights. Appendix A.3 gives the memory sampling intervals. Appendix B.4 explains how future query kk is supervised with the label at t+kΔt+k\Delta. We collect them in one table below.

SettingBlock size BBFuture queries KKMemory sampling intervalλsub\lambda_{sub}λobj\lambda_{obj}
RoboCasa36581616 frames / 0.8 s0.250.25
RoboCasa-Long81616 frames / 0.8 s1.001.00
Real robot81630 frames / 1.0 s0.750.75

Here, BB is the number of history tokens in each block-causal attention block. KK is the number of future queries. Query kk is supervised with the label at t+kΔt+k\Delta. The sampling interval Δ\Delta determines how often a multi-view observation is compressed into one memory token. It is separate from the control frequency.

W4: One-token compression and spatial information

The one-token compression applies only to historical memory. The current multi-view observation is still processed by the vision-language model (VLM) and supplied to the Action Expert for spatial control. The history token is designed to retain persistent semantic state, such as whether an object has been moved, grasped, or placed. It is not intended to reconstruct the exact spatial layout of an earlier frame.

The probe experiment tests whether task-state information remains decodable after compression. It reaches 69.98% and 68.22% on the two RoboCasa365 splits and 85.62% on RoboCasa-Long. A single cached token therefore retains substantial information about the corresponding task state.

The single-token design also controls the cost of long histories. On RoboCasa-Long, end-to-end inference takes 764.52 ms and 10.21 GB. These measurements include all three memory-view encodings and the memory-to-action layers. From 1k to 8k steps, graphics processing unit (GPU) memory increases by only 0.08 GB and latency does not increase with history length. One token per sampled timestep and block-wise caching keep the memory cost stable over multi-minute execution.

AuthorsOpenReview ↗

Response to Reviewer 8jLA Part 3

W5A: Scope of RoboCasa-Long and generalization

RoboCasa-Long is designed for natural multi-minute household tasks. Existing benchmarks provide many short or purpose-built memory tasks, but few natural household activities that span several work areas and require minutes of continuous execution. RoboCasa-Long provides these tasks, 200 demonstrations per task, and progress-aware evaluation for the community.

RoboCasa-Long is designed to isolate long-horizon execution under controlled task settings. It is not intended as a held-out-kitchen or held-out-object benchmark. We evaluate scene and object generalization on the full RoboCasa365 suite. Across Atomic, Composite-Seen, and Composite-Unseen, the training and evaluation episodes vary kitchen scenes, object categories, and object placements. Composite-Unseen additionally tests unseen task compositions. The two benchmarks therefore serve different roles: RoboCasa-Long tests multi-minute memory, while RoboCasa365 evaluates generalization across scenes and objects.

W5B: Existing memory baselines

Our original baselines isolate the source of the gain: XR-1 measures the effect of adding memory to the same backbone, while MOST-Raw keeps the same history and architecture but removes task-state supervision. To further compare with an existing memory design, we directly train HAMLET, a history-aware vision-language-action (VLA) policy [2], unchanged on all three RoboCasa-Long tasks. It uses the same demonstrations, training budget, and 30-rollout-per-task protocol. The table reports averages over all three tasks.

MethodSRMC@50MPOPSUPS
XR-10.003.3320.8314.9230.55
MOST-Raw0.0020.0028.7522.7146.16
HAMLET0.0017.7725.7420.9634.23
MOST6.6634.4442.2238.7776.17

MOST outperforms HAMLET on every reported metric. It improves MP by 16.48 points and UPS by 41.94 points across the complete RoboCasa-Long benchmark. The RMBench table above additionally provides comparisons with MemoryVLA, MemER, Mem-0, and EventVLA.

W5C: Status of the GR00T N1.5 numbers

GR00T N1.5 follows the same official RoboCasa365 protocol used in our evaluation. The training-data splits, target tasks, and evaluation scenes are the same. GR00T therefore provides a fair benchmark-level memoryless reference and is not used in the controlled memory ablation.

W5D: Fixed-base real-robot evaluation

The real-robot base is intentionally fixed to isolate memory-dependent counting and stopping from navigation failures. The robot rotates its waist between the pickup and placement workspaces. This preserves repeated viewpoint and workspace changes while preventing navigation errors from affecting the memory comparison. The real-robot experiment evaluates memory under physical perception and control. Mobile navigation and manipulation are evaluated in simulation.

References

[1] Sridhar, A., et al. “MemER: Scaling Up Memory for Robot Control via Experience Retrieval.” ICLR, 2026. arXiv:2510.20328.

[2] Koo, M., et al. “HAMLET: Switch Your Vision-Language-Action Model into a History-Aware Policy.” ICLR, 2026. arXiv:2510.00695.

[3] Wu, Y., et al. “Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies.” arXiv:2610.14406, 2026.

I thank the authors for their detailed responses! After reading them in detail, I decided to maintain my positive score.

Reviewer_77oMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript studies task-relevant memory for long-horizon robotic manipulation. It compresses multi-view observations into temporal tokens and applies task-progress and object-state supervision to both historical and predictive tokens, which condition action generation. It also introduces RoboCasa-Long, comprising three multi-minute household tasks. Validation covers RoboCasa365, the new benchmark, and real-world counting tasks, using comparisons with memory-free and matched visual-history policies, component ablations, and progress-aware metrics.

Critical strengths
  1. The proposed memory design improves average success from 68.0% to 72.1% over the matched visual-history baseline on RoboCasa365 Target-100%. This comparison supports its value beyond simply adding historical observations, while the component ablations provide further insight into the contributions of task-progress and object-state supervision.
Critical weaknesses
  1. This manuscript lacks sufficient comparisons with existing memory-based VLA methods. Long-horizon manipulation is precisely where memory matters, making these methods natural baselines. It remains unclear when and why explicit task-progress and object-state supervision offers an advantage over alternative memory mechanisms, particularly after accounting for annotation costs and computational overhead. Comparisons with memory-free policies and internal ablations show that the components help, but do not establish that this approach is preferable to existing alternatives.

  2. Although this manuscript describes block-wise caching and the removal of auxiliary heads at inference, I could not find measurements of training time or inference latency. I would like to see these costs quantified, especially as the memory history grows, to better assess the practical trade-off.

Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 77oM Part 1

Response summary

Thank you for recognizing that the matched MOST-Raw comparison already shows the value of MOST beyond simply adding visual history, and that the component ablations support the roles of object-state and task-progress supervision. This is the central claim of the paper: long-horizon memory should preserve persistent task state rather than merely accumulate visual observations. The remaining concerns ask whether this advantage holds against existing memory methods and what it costs in annotation and computation. These concern empirical coverage and practical characterization rather than a change to the core method. We address them with a directly trained HAMLET baseline, RMBench results against several memory policies, a clearer analysis of scope and annotation cost, and detailed training and inference measurements.

W1A: Comparison with existing memory-based vision-language-action models

Our baselines are chosen to separate the source of the gain. Xiaomi-Robotics-1 (XR-1) measures the effect of adding memory to the same backbone, while MOST-Raw keeps the same history and architecture but removes task-state supervision. To further compare with an existing memory design, we add HAMLET, a history-aware vision-language-action (VLA) policy [1]. We train it unchanged on all three RoboCasa-Long tasks using the same demonstrations, training budget, and 30-rollout-per-task protocol. The table reports averages over all three tasks.

Metrics follow Appendix B.5: success rate (SR), milestone coverage at 50% progress (MC@50), max progress (MP), ordered progress score (OPS), and unordered progress score (UPS).

MethodSRMC@50MPOPSUPS
XR-10.003.3320.8314.9230.55
MOST-Raw0.0020.0028.7522.7146.16
HAMLET0.0017.7725.7420.9634.23
MOST6.6634.4442.2238.7776.17

MOST outperforms HAMLET on every reported metric. It improves MP by 16.48 points and UPS by 41.94 points. This provides a direct comparison with an existing history-aware VLA across the complete RoboCasa-Long benchmark.

We also evaluate MOST on two representative tasks from the Memory-Dependent Robotic Manipulation Benchmark (RMBench). Battery Try requires memory of repeated state-changing attempts. Press Button requires count-based task-progress tracking. The published comparison includes EventVLA, MemER [2], Mem-0, HAMLET [1], and MemoryVLA.

MethodBattery TryPress ButtonTwo-task Avg.
π0.5\pi_{0.5}†30316.5
Uniform Sampling†28315.5
EventVLA†35319.0
MemER†27013.5
Mem-0†28014.0
HAMLET†26013.0
MemoryVLA†33016.5
MOST35520.0

† Published RMBench results reported by IMPRINT [3]. MOST is evaluated with 100 rollouts per task using the RMBench success criteria.

We report these results as a two-task subset rather than the complete M(n)M(n) suite. MOST matches the best published Battery Try result and achieves the highest Press Button result and two-task average. These results provide comparisons with HAMLET, MemoryVLA, MemER, Mem-0, and EventVLA on an external memory benchmark.

W1B: When and why explicit task-state supervision helps

Explicit task-state supervision is most useful when a long task evolves through persistent world states and execution stages. We therefore evaluate it on RoboCasa365 and RoboCasa-Long, which contain natural household tasks where the policy must remember which objects have been moved and which subtasks have been completed. We also use RMBench counting tasks to test whether the policy can retain accumulated task progress outside RoboCasa. Together, these benchmarks cover the two forms of information targeted by MOST: object-state transitions and task progress.

BenchmarkMetricXR-1MOSTGain
RoboCasa365 Target-100%Average SR65.972.1+6.2
RoboCasa-LongMP20.8342.22+21.39
RoboCasa-LongUPS30.5576.17+45.62

MOST improves both benchmarks, with substantially larger gains on the multi-minute RoboCasa-Long tasks. As the task becomes longer, more completed subtasks and persistent object changes must survive in memory. Explicit task-state supervision gives the memory token a stable target for this information instead of requiring the policy to recover it from changing visual observations.

This design also has a clear boundary. MOST is intended for tasks whose relevant history can be summarized by object states and task progress. Some RoboMME tasks instead require exact recall of visual content from an earlier observation, such as remembering a demonstrated insertion pattern. These tasks require detailed visual memory and are outside the intended scope of MOST.

AuthorsOpenReview ↗

Response to Reviewer 77oM Part 2

W1C: Annotation cost

All large-scale task-progress and object-state labels in RoboCasa365 and RoboCasa-Long are exported automatically from simulator states and task definitions. They require no manual annotation. The small real-robot dataset is labeled manually because simulator state is unavailable. Its closed-set subtask and object-state labels keep this cost controlled.

The supervision therefore adds little annotation cost at the scale used for training. It improves RoboCasa365 average SR from 68.0% for MOST-Raw to 72.1%. On RoboCasa-Long, it improves MP from 28.75% to 42.22% and UPS from 46.16% to 76.17%.

Providing high-quality task-progress and object-state annotations is also part of our contribution to the community. These annotations make it possible to study memory on natural multi-minute manipulation tasks rather than only on short tasks or manually constructed memory tests. Since the large-scale simulation labels are generated automatically, they do not create a manual annotation bottleneck. They are a reusable data contribution rather than a practical limitation of the method.

W2: Training and inference overhead

We report the training and inference cost below. All measurements use bfloat16 (BF16) on NVIDIA A800 80 GB graphics processing units (GPUs) under identical settings. Inference uses one GPU and training uses eight GPUs. The auxiliary heads are removed during inference.

BenchmarkModelParamsLatencyInference GPUTraining TimeTraining GPU
RoboCasa365XR-15.1B651.44 ms9.68 GB43.43 h~54 GB/GPU
RoboCasa365MOST5.3B783.69 ms10.16 GB58.43 h~58 GB/GPU
RoboCasa-LongXR-15.1B653.72 ms9.68 GB23.51 h~54 GB/GPU
RoboCasa-LongMOST5.3B764.52 ms10.21 GB31.40 h~58 GB/GPU

The Memory Expert adds 0.2B parameters. On RoboCasa365, it adds 132.25 ms and 0.48 GB during inference, while average SR improves from 65.9% to 72.1%. On RoboCasa-Long, it adds 110.80 ms and 0.53 GB, while MP improves from 20.83% to 42.22% and UPS from 30.55% to 76.17%.

Training takes 15.00 additional hours on RoboCasa365 and 7.89 additional hours on RoboCasa-Long, with approximately 4 GB/GPU additional training memory. The extra training time includes decoding and loading historical video frames. It is therefore not caused only by the 0.2B parameter increase.

We also measure inference latency and GPU memory every 1k steps over an 8k-step RoboCasa-Long rollout.

MetricModel1k2k3k4k5k6k7k8k
Latency (ms)XR-1662.90649.32655.22650.76659.50644.75664.67642.69
Latency (ms)MOST771.58774.63761.04758.45756.24763.46792.52737.88
GPU memory (GB)XR-19.689.689.689.689.689.689.689.68
GPU memory (GB)MOST10.1310.1410.1610.1710.1810.1910.2010.21

MOST latency does not increase with history length. GPU memory grows by only 0.08 GB from 1k to 8k steps. Compressing each sampled multi-view observation into one token and caching completed blocks prevents the model from repeatedly processing the full visual history.

References

[1] Koo, M., et al. “HAMLET: Switch Your Vision-Language-Action Model into a History-Aware Policy.” ICLR, 2026. arXiv:2510.00695.

[2] Sridhar, A., et al. “MemER: Scaling Up Memory for Robot Control via Experience Retrieval.” ICLR, 2026. arXiv:2510.20328.

[3] Wu, Y., et al. “Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies.” arXiv:2610.14406, 2026.

Dear Reviewer 77oM,

As the ICLR 2027 discussion period approaches its end, we would greatly appreciate it if you could review our rebuttal and additional results. We have expanded the comparisons with an existing memory-based policy and an external memory benchmark, and provided further clarification on annotation and computational costs.

We would be happy to clarify any remaining concerns or questions.

Thank you for your time and consideration.

Best regards,
Authors

Reviewer_NH4HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that long-horizon policy memory should be organized around what information needs to survive, and proposes MOST, an end-to-end policy built on XR-1. In the framework, a memory Expert compresses multi-view history into one token per timestep, a learned view query fuses the DINOv2 features of each view. Learned future queries are appended, and the sequence is processed with block-causal attention. In addition, the authors also introduce RoboCasa-Long, a benchmark of three multi-minute mobile manipulation tasks with progress-aware metrics. Across RoboCasa 365 and real-world evaluations, MOST improves over XR-1, an existing memoryless baseline.

Critical strengths

Strengths

  • The paper is easy to follow. Framing memory around task-relevant state (object states and task progress) rather than raw visual history is well-motivated.
  • RoboCasa-Long is useful contributions for more reliable evaluation of long-horizon policies.
  • The long-horizon analyses (e.g., horizon-grouped analysis, representation study) are thorough.
Critical weaknesses

Weaknesses

[W1] Limited baselines, backbones, and benchmarks. MOST is compared only with its own backbone XR-1 and its own variant MOST-Raw, plus published GR00T N1.5 numbers. No existing memory-augmented VLA is evaluated, even though several are discussed in the related work. Beyond the proposed RoboCasa-Long, evaluation is limited to RoboCasa365. MOST is not tested on existing memory benchmarks (e.g., RoboDojo) or on other standard manipulation benchmarks (e.g., LIBERO).

[W2] Marginal gain from combining the two supervision signals. The paper’s central claim is that memory should encode both object states and task progress. However, Table 4 shows that most of the improvement over visual history is already obtained with either signal alone. Moreover, this ablation is conducted only on RoboCasa365 Composite-Seen.. As a result, it remains unclear whether both signals are truly necessary.

Questions for authors

Questions

[Q1] How does MOST compare with existing memory-augmented VLAs (e.g., MemoryVLA, MemER, HAMLET) on established memory benchmarks (e.g., RoboMME, RoboDojo)?

[Q2] Compared to existing VLAs, does MOST maintain comparable performance on standard non-memory benchmarks (e.g., SIMPLER, LIBERO)?

[Q3] Are both task-progress and object-state supervision necessary across different benchmarks to ensure performance gain?

[Q4] What is the additional cost of the Memory Expert in terms of inference latency, memory footprint, and training time, relative to XR-1?

5 comments
AuthorsOpenReview ↗

Response to Reviewer NH4H Part 1

Response summary

Thank you for recognizing that organizing memory around task-relevant object states and task progress is well motivated, that RoboCasa-Long is a useful contribution, and that our long-horizon analyses are thorough. These points capture the two central contributions of the paper: MOST provides compact state-oriented memory for natural long-horizon manipulation, and RoboCasa-Long provides multi-minute tasks with progress-aware evaluation. The remaining questions concern broader empirical coverage, the complementarity of the two supervision signals, and computational cost. They ask for stronger evidence around the existing claims rather than a different method or motivation. We address them with a directly trained HAMLET baseline, an external memory benchmark, RoboCasa-Long supervision ablations, and detailed efficiency measurements.

W1/Q1: Comparison with existing memory-augmented vision-language-action models

Our baselines are chosen to separate the source of the gain. Xiaomi-Robotics-1 (XR-1) measures the effect of adding memory to the same backbone, while MOST-Raw keeps the same history and architecture but removes task-state supervision. To further compare with an existing memory design, we add HAMLET, a history-aware vision-language-action (VLA) policy [1]. We train it unchanged on all three RoboCasa-Long tasks using the same demonstrations, training budget, and 30-rollout-per-task protocol.

Metrics follow Appendix B.5: success rate (SR), milestone coverage at 50% progress (MC@50), max progress (MP), ordered progress score (OPS), and unordered progress score (UPS).

MethodSRMC@50MPOPSUPS
XR-10.003.3320.8314.9230.55
MOST-Raw0.0020.0028.7522.7146.16
HAMLET0.0017.7725.7420.9634.23
MOST6.6634.4442.2238.7776.17

MOST outperforms HAMLET on every reported metric. It improves MP by 16.48 points and UPS by 41.94 points. This provides a direct comparison with an existing history-aware VLA across the complete RoboCasa-Long benchmark.

W1/Q1/Q2: Evaluation beyond the proposed benchmark

Our benchmarks also serve complementary roles. RoboCasa365 tests standard manipulation and scene/object generalization, RoboCasa-Long tests natural multi-minute execution, and the real-robot tasks test counting memory under physical perception and control. To add a standard external memory benchmark, we evaluate two RMBench tasks aligned with MOST's target information: Battery Try tests persistent state history, while Press Button tests count-based task progress. We report them as a two-task subset.

MethodBattery TryPress ButtonTwo-task Avg.
π0.5\pi_{0.5}†30316.5
Uniform Sampling†28315.5
EventVLA†35319.0
MemER†27013.5
Mem-0†28014.0
HAMLET†26013.0
MemoryVLA†33016.5
MOST35520.0

† Published RMBench results reported by IMPRINT [3].

MOST matches the strongest published result on Battery Try, achieves the highest result on the count-dependent Press Button task, and obtains the highest two-task average. This provides external evidence that task-state memory generalizes beyond RoboCasa and remains competitive with existing memory-augmented VLAs.

For standard manipulation, RoboCasa365 provides a 50-task benchmark covering 18 Atomic tasks and 32 Composite-Seen/Unseen tasks. The Atomic split includes fundamental manipulation skills such as pick-and-place, opening and closing doors, and opening drawers. On Target-100%, MOST improves average SR from 65.9% for XR-1 and 68.0% for MOST-Raw to 72.1%, showing that the memory mechanism retains strong performance on standard manipulation skills while improving composite long-horizon execution.

W2/Q3: Complementarity of object-state and task-progress supervision

We extend the object-state and task-progress ablation from RoboCasa365 to the substantially longer RoboCasa-Long benchmark.

Memory supervisionSRMC@50MPOPSUPS
None, XR-10.003.3320.8314.9230.55
Raw visual history0.0020.0028.7522.7146.16
Object state only1.1123.3331.8128.0554.36
Task progress only2.2230.0036.9535.2369.00
Object state + task progress6.6634.4442.2238.7776.17

Each signal independently improves over raw visual history. Their combination is consistently strongest, improving over the stronger task-progress-only variant by 4.44 points in SR, 5.27 in MP, 3.54 in OPS, and 7.17 in UPS. Together with the RoboCasa365 ablation in Table 4, these results show that object-state and task-progress supervision provide complementary information: one preserves persistent world changes, while the other tracks execution stage.

AuthorsOpenReview ↗

Response to Reviewer NH4H Part 2

Q4: Computational overhead of the Memory Expert

We report the training and inference cost below. All measurements use bfloat16 (BF16) on NVIDIA A800 80 GB graphics processing units (GPUs) under identical settings. Inference profiling uses one GPU, while training uses eight GPUs. The auxiliary heads are removed during inference.

BenchmarkModelParamsLatencyInference GPUTraining TimeTraining GPU
RoboCasa365XR-15.1B651.44 ms9.68 GB43.43 h~54 GB/GPU
RoboCasa365MOST5.3B783.69 ms10.16 GB58.43 h~58 GB/GPU
RoboCasa-LongXR-15.1B653.72 ms9.68 GB23.51 h~54 GB/GPU
RoboCasa-LongMOST5.3B764.52 ms10.21 GB31.40 h~58 GB/GPU

The Memory Expert adds only 0.2B parameters. On RoboCasa365, the additional cost is 132.25 ms and 0.48 GB of inference memory, while average SR improves from 65.9% to 72.1%. On RoboCasa-Long, the additional cost is 110.80 ms and 0.53 GB, while MP improves from 20.83% to 42.22% and UPS from 30.55% to 76.17%. Training takes 15.00 additional hours on RoboCasa365 and 7.89 additional hours on RoboCasa-Long, with approximately 4 GB/GPU additional training memory. The extra training time also includes decoding and loading the historical video frames used by the memory module, so it is not caused only by the 0.2B parameter increase.

We also profile latency and GPU memory every 1k steps over an 8k-step RoboCasa-Long rollout:

MetricModel1k2k3k4k5k6k7k8k
Latency (ms)XR-1662.90649.32655.22650.76659.50644.75664.67642.69
Latency (ms)MOST771.58774.63761.04758.45756.24763.46792.52737.88
GPU memory (GB)XR-19.689.689.689.689.689.689.689.68
GPU memory (GB)MOST10.1310.1410.1610.1710.1810.1910.2010.21

MOST latency fluctuates within a narrow range rather than increasing with history length, while GPU memory grows by only 0.08 GB from 1k to 8k steps. Compressing each sampled multi-view observation into a single temporal token, together with block-wise caching, keeps both latency and memory bounded without repeatedly processing the full raw visual history.

References

[1] Koo, M., et al. “HAMLET: Switch Your Vision-Language-Action Model into a History-Aware Policy.” ICLR, 2026. arXiv:2510.00695.

[2] Sridhar, A., et al. “MemER: Scaling Up Memory for Robot Control via Experience Retrieval.” ICLR, 2026. arXiv:2510.20328.

[3] Wu, Y., et al. “Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies.” arXiv:2610.14406, 2026.

Dear Reviewer NH4H,

As the ICLR 2027 discussion period approaches its end, we would greatly appreciate it if you could review our rebuttal and additional results. We have addressed your questions with an existing memory baseline, an external memory benchmark, additional supervision ablations, and computational-cost measurements.

We would be happy to clarify any remaining concerns or questions.

Thank you for your time and consideration.

Best regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Chairs and Reviewers,

We thank the reviewers for the detailed discussion and summarize the paper’s contribution together with the completed rebuttal evidence.

MOST makes a focused and efficient representational contribution to long-horizon robot memory. It addresses what task-relevant information should persist after observations leave the current view. MOST explicitly trains policy-internal memory to preserve object states and task progress, compresses each sampled multi-view observation into one token, and directly conditions action generation. We also introduce RoboCasa-Long, a benchmark of natural multi-minute household tasks with progress-aware evaluation.

The controlled comparisons isolate task-state representation from the general effect of adding history. MOST-Raw shares the same history, compression mechanism, architecture, and training setting while removing task-state supervision. MOST improves over MOST-Raw from 68.0 to 72.1 average success on RoboCasa365. On RoboCasa-Long, it improves maximum progress from 28.75 to 42.22 and unordered progress from 46.16 to 76.17. Additional controls show that arbitrary auxiliary supervision is insufficient and that object-state and task-progress signals are complementary. The larger long-horizon gains indicate that structured memory becomes particularly valuable as persistent changes and completed stages accumulate.

The benefit extends beyond the original benchmark without sacrificing standard manipulation. MOST outperforms the directly trained HAMLET baseline across all RoboCasa-Long metrics. On the complete external RMBench M(n)M(n) category, it improves matched XR-1 from 27.75 to 51.00. On all four LIBERO suites, it achieves 98.2 average success.

The memory is both informative and practical. Finalized tokens reach 85.62% probe accuracy on RoboCasa-Long, and learned non-zero gates show that the action pathway uses them. Over an 8k-step rollout, latency does not grow with history length, while GPU memory increases by only 0.08 GB from 1k to 8k steps.

The scope is specific. MOST targets tasks governed by persistent world states and execution progress. Tasks requiring exact visual or motion recall may favor perceptual or episodic memory. The contribution is therefore a controlled study of an important memory representation, not a claim that one mechanism fits every memory-dependent task.

Together, MOST and RoboCasa-Long provide a method, controlled evidence, and a community benchmark for preserving task-relevant information throughout long-horizon manipulation.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.